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CHAPTER 1 
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The primary purpose of ICILS 2018 is to assess systematically the capacities of students to use 
ICT productively for a range of different purposes, in ways that go beyond a basic use of ICT. 


ICILS 2018 includes authentic computer-based assessments that are admini 


stered to students 


in their eighth year of schooling. These generate data reflecting two dimensions of ICT-related 


capacities: 
First, 


ICILS 2018 assesses CIL. This was first measured in ICILS 2013, wh 


as “an individual’s ability to use com 
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uters to investigate, create, and comm 
participate effectively at home, at school, in the workplace, and in society” ( 


ere it was defined 
unicate in order to 
Fraillon et al. 2013 


p. 17). ClL refers to a student's ability to use computer technologies to collect and manage 


information and to produce and ex 
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nd, ICILS 2018 assesses CT, w 


change information. 


hich is the type of thinking used when programming a 


computer or developing an application for another type of digital device. We define CT to 
refer to an individual's ability to recognize aspects of real-world problems which are appropriate 
for computational formulation and to evaluate and develop algorithmic solutions to those problems 
so that the solutions could be operationalized with a computer. Yadav et al. (2018, pp. 91-92) 
articulated the relationships between CT and computer science, defining CT as being focused 
on the processes of “abstraction, algorithms and automation.” 
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ICILS 2018 investigates variations in CIL and CT between and within countries and the 
relationships between those constructs and student attributes (background characteristics and 
developed attributes) including their use and experience of computer technologies. ICILS 2018 
also investigates how CT is related to CIL. 


A secondary purpose of ICILS 2018 is to investigate the use of computers and other digital 
devices by students and teachers, as well as their attitudes toward the use of computer 
technologies. Some of these aspects of computer use are directly related to student outcomes, 
while others may not be directly associated with them but may inform our understanding of 
the broader context in which computer technologies are used both within and outside school. 
Further contextual information is provided by schools and education systems about the policies, 
resources, and pedagogies regarding computer technologies. 


1.3. Purpose of the ICILS assessment framework 


The ICILS assessment framework articulates the basic structure of the study. It provides a 
description of the field and the constructs to be measured. It outlines the design and content 
of the measurement instruments, sets down the rationale for those designs, and describes how 
measures generated by those instruments relate to the constructs. In addition, it hypothesizes 
relations between constructs so as to provide the foundation for some of the analyses that 
follow. Above all, the framework links ICILS to other work in the field so that the contents of this 
assessment framework combine theory and practice in an explication of “both the ‘what’ and the 
‘how’” (Jago 2009, p. 1) of ICILS. 


1.4 Background to the study 


Schools have been using computers in education and teaching about computing forapproximately 
four decades, and it is a field that remains under active development. One of the important 
changes in this field has been a shift in the focus on technical usage and programming, including 
simplified programming using languages such as Logo (McDougall et al. 2014), to the widespread 
use of applications incorporating information management and communications. Punter et al. 
(2017) argued that the widespread use of the internet, as well as the ready availability of office 
applications, changed the nature of computer use. Caeli and Bundsgaard (2019) identified 
four phases of computer use in education in Denmark, beginning with exploring implications of 
computing for society in combination with aspects of CT, progressing through a phase in the 
1990s that emphasized enabling students to use computer applications, and a phase in the early 
2000s that focused on the pedagogical use of digital resources, culminating in the most recent 
phase, which focuses on CT. 


ICILS 2013 conceptualized CIL in terms of two strands that framed the skills and knowledge 
addressed by the instruments (Fraillon et al. 2013, pp. 34-35). Strand 1 focused on the receptive 
and organizational elements of information processing and management (understanding 


computer use and accessing, evaluating and managing digital information) and strand 2 was 


concerned with producing and exchanging information (transforming, creating and sharing 
computer-based information). The assessment consisted of four 30-minute modules. A module 


was a set of questions and tasks based on a real-life theme and following a linear narrative 


structure. Each module had a series of small discrete tasks (skill execution and information 


management) followed by a large task that required the use of several applications to produce 


an information product that was scored by trained scorers according to specified scoring rubrics. 
The results of ICILS 2013 indicated that one dimension, CIL, underpinned the responses to the 
assessment (Gebhardt and Schulz 2015). 


As a result of research and development associated with computing in schools, changes in 
digital technologies themselves, and changes in conceptions of the meaning of digital capability, 


there has been an emerging interest in computer science (CS) and computational thinking (CT). 
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as a concept that embraces problem solving and system design, and 
tral to computer science. This concept includes the ways of thinking 
puter as part of computer literacy (Grover and Pea 2013; Lye and Koh 


2014). Computational thinking can be seen as “applying tools and techniques from computer 
science to understand and reason about both natural and artificial systems and processes” 


Royal Society 2012, p. 29). Shute et al. 
problems algorithmically (with or without the assistance of computers) by applying solutions 
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2017, p. 142) argued that CT is required to solve 


performance-based asses 


that are reusable in different contexts. They elaborated that CT is “a way of thinking and acting, 
which can be exhibited through the use of particular skills, which then can become the basis for 


the capturing of the data 


sments of CT skills.” They suggested that CT involves six elements: 


decomposition, abstraction, algorithm design, debugging, iteration, and generalization. CT does 
not necessarily involve developing or implementing a formal computer code (Barr et al. 2011). 
However, assessments of CT are typically set in computer environments because those facilitate 


that reflect the steps in problem solving. These steps usually involve 


developing or assembling instructions (often including blocks of code) that are necessary to 


accomplish a task (Brennan and Resnick 2013). 


Learning to use and appl 
education (Peyton Jones 


y computer science is regarded as an important element of school 
2011). The UK’s Royal Society (2017) advocated for an increased 


attention to computer science in the British school curricula and emphasized the importance of 


computational thinking as 


part of digital literacy. Wing (2006, p. 33) argued that the concept of 


CT is applicable to all individuals rather than just computer scientists. 
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3) developed the curriculum resource Exploring Computer Science (ECS) 
ning of CT. This curriculum package includes resources, lesson plans, 
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Grover et al. 2015; Grover 2017). PACT is based on design patterns for major CT practices and 
involves judging the quality of the instructions (or coding steps) that have been assembled. 


There have also been other approaches to the assessment of CT. Chen et al. (2017) developed 
an instrument for primary school students to assess CT that was based on coding in robotics 
and reasoning of everyday events, and linked to a “robotics curriculum.” Zhong et al. (2016) 
developed a three-dimensional assessment framework based on the concepts of directionality, 
openness and process. The assessment included three pairs of tasks that were based on a three- 
dimensional programming language: (1) closed forward tasks and closed reverse tasks, (2) semi- 
open forward tasks and semi-open reverse tasks, and (3) open tasks with a creative design report 
and open tasks without a creative design report. Students’ codes were assessed by the research 
team based on sets of rubrics reflecting elements of CT. They concluded that semi-open tasks 
were more discriminating than others, but that a combination of tasks was needed to assess 
the various elements of CT. What appear to be common elements in assessments of CT are the 
capturing of instructions developed by students (almost always using a computer environment) 
and the judging of the quality of those instructions against a set of criteria reflecting aspects of CT. 


1.7 Recent education policy developments related to CIL 


and CT 


Since ICILS 2013 there have been a number of developments in education policy concerned 
with computer and information literacy. The participating national research centers provided 
information about these developments, together with examples of research publications 
connected to these developments. In this section, we focus on those developments that relate to 
ICILS 2018. Two common themes for developments in educational policy are an enhanced role 
for CT and an interdisciplinary view of the place of ICT in education. 


Areview conducted by the European Commission highlighted the development of CT and CS as 
part of compulsory education (Bocconi et al. 2016), and noted the variations amongst European 
countries in integrating CT into compulsory education. The review emphasized the need for 
research on the pedagogies and assessment methods associated with the implementation of CT, 
as well as research on the results of the implementation. It also identified a need for teacher 
support through large-scale professional learning opportunities, such as those initiated in 
England, and the need to develop new assessment tools and methods. 


One review characterized the curriculum in Finland 
combine knowledge and skills with attitudes across disciplines (Kwon and Schroderus 2017). 
In the curriculum introduced in 2016, ICT is defined as a transversal skill that crosses subject 
areas. It concerns operating principles and concepts (including CT), responsible safe use of ICT, 
application in information management, and creative work and interaction and networking 
(Finnish National Agency for Education 2016). A country report on ICT in education in Finland 
included many other aspects, such as the need for teacher education and the development 
of digital curriculum content (Koskinen 2017). Saari and Santti (2018) pointed to the priority 
accorded to digitalization through modernizing infrastructure and expanding teacher education, 
and how the reform conflicts with traditions of devolved authority. Kaarakainen et al. (2018) 
reported on performance-based testing of students’ and teachers’ ICT skills, and argued that 
assessment data are pivotal to improving the computer literacy of students in Finland. 
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The 16 German federal states developed a national competence framework and strategy for 
developing digital competences (KMK [Kultusministerkonferenz] 2016). This framework was not 
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ICT use in mathematics was spread evenly across exploring principles and concepts, practicing 
skills and procedures, looking up ideas and information, and processing and analyzing data, 
while, in science lessons, ICT was more frequently used for looking up ideas and information. 


More than half of the grade 8 students surveyed in TIMSS 2015 used the internet to access 


. 
students (Martin et al. 2016). 


The O 
indicated that, across O 
an average of 0.8 computers 
to the internet (O 
in OECD countries reported 
to simulate natural or technical 


ECD report for the Progra 


regu 


processes (O 


shown that computer and internet 
grou 


associated with student bac 


A survey of ICT i 
on in 2013 inclu 
ISCED; see UN 
evel 2 (grade 8 
had been using 
based activities 
Commiss 
particularly wel 
simulations). One-third of the st 
resources at least once a week. 


n school ed 
ded 
ESCO [United 


com 


ucation commissio 
a survey of 


European Commission 2013). 
puters for m 
more frequently at home than at school. Students surveyed in 
ion study rarely reported using applications 
suited to ICT use during lessons (such as data-logging tools and co 
used digital textbooks a 


ations 


ned by the 
students at International Stand 
Education, Scientific and Cultural Organ 


mme for International Student Assessment (PISA) in 
ECD countries in schools with enrolled 15-year-old students, there was 
per student, and that almost all of these computers were connected 
ECD 2016b, p. 190). However, less than one-tenth of 15-year-old st 
ar or very frequent use of computers or virtual laboratories 
ECD 2016b, p. 119). Data from ICILS 2013 have 
access at school varies across and 
nd and school contexts (Fraillon et al. 2014). 


European Commission and re 


formation and resources, and more than two-thirds used the internet to collaborate with other 


2015 


udents 


within countries, and is also 


ported 
Education 
ization] 2011) 


ard Classification of 


if 


udents stated that they 


the amount of student-centered 


The report provided evid 


learning and frequency 


uropean Commission survey al 


so identified three grou 


” 


the report's authors termed “fun 
videos), “ 


— 
Wy 


competences when they [had] 
Commission 2013, p. 15). Conf 


The report's authors also indi 


Eighty percent o 
ore than four years. Students also reported undertaking ICT- 


ps of 


(e.g., streaming or download 
earning’ (e.g., online news, information searching, and 
This classification, however, did not include activities involvi 
oftware applications) for school-related document preparatio 


cated that students were 


that t 


the grade 8 students said they 


Eu 


the European 
m considered 
mputer 
nd multimedia 


he research tea 


ence of a positive association between 
of ICT 


use for classroom activities. The 
CT-based activities at home that 
ing multimedia, music, movies, or 
earning programs), and “games.” 
ng the use of computer utilities 
n. 


nfident in thei 


high 


access to/use of ICT at home and at school” ( 
dent students also tended to 


m 


r “digital 
European 


ore CO 


ICT on their work and leisure. Th 


be positive about the impact of 


e results from this study suggest that pedagogical use of ICT is 


not associated with more abund 


because the extents and patter 


analyses showed that, after controlli 


or socioeconomic status, studen 


use at home were positively associated wi 


ant 


ICT u 


CT resourcing; despite enhanced resourcing in 
years before the study, school use of ICT had not increased since 2006. 


ICILS 2018 has a specific interest in 
ns of 


data concerning computer use by students a 
se are potentially related to ClL and CT. 


the several 


nd teachers 
ultivariate 


ng fo 


r the effect of background variables suc 


h as gender 


t experien 


ce of computer use and th 
th ClL scores in most countries (Fraillon et al. 2014). 


eir frequency of computer 


Student access to a home internet connection and the number of computers at home also had 


statistically significant associations with C 


L scores in about half of t 


he participating education 


systems. Greater interest in and enjoyment of ICT use was associated with higher CIL scores 


in nine of the 14 countries that met the ICILS sampling requireme 


education systems, there was evidence of an association between C 


nts. In addition, in several 
L scores and the extent to 


which students reported having learned about |CT-related tasks at school. 


In ICILS 2013, CIL was also positively associated with basic ICT skills self-efficacy (student 
confidence in undertaking basic ICT-based tasks such as creating or editing documents or 
searching and finding information on the internet), but not with advanced ICT self-efficacy (student 
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confidence to carry out tasks such as building or editing a web page or creating a computer 
program or macro) (Fraillon et al. 2014; Rohatgi et al. 2016). This finding is congruent with the 
nature of the CIL construct, which is made up of information literacy and communication skills 
that are not necessarily related to advanced computer skills such as programming or database 
management. Even though CIL is computer-based, in the sense that students demonstrate CIL 
in the context of computer use, the ClL construct itself does not emphasize high-level computer- 
based technical skills. 


Data from ICILS 2013 have also been used in secondary analyses to investigate the ways in which 
aspects of computer use by teachers and students are related to students’ CIL (Bundsgaard and 
Gerick 2017; Gerick 2018). However, ICILS 2018 is also interested in describing variations in 
computer use as part of a broader understanding of the roles of information technologies in schoo 
education. Secondary analyses of ICILS 2013 suggest that teacher attitudes are associated with 
the extent and the ways that they use ICT in their teaching (Drossel et al. 201 7a; Eickelmann and 
Vennemann 2017). Similar analyses investigated the influence of professional development on 
teacher attitudes to, and use of, ICT in school education (Drossel and Eickelmann 201 7). In fact, 
there is evidence that school factors, including teachers’ collaborative use of ICT, can shape the 
pedagogical use of ICT for teaching purposes (Drossel et al. 2017b; Gerick et al. 2017). ICILS 
2018 provides updated information on these influences, and extends them to the teaching of CT, 
based on representative samples of grade 8 students and teachers of grade 8 students across a 
range of different education systems. 


1.9 Research questions 


ICILS aims to investigate the extent of ClLand CT among grade 8 students, and the associations 
of these learning outcomes with student background, developed attributes, experience with 
using computer technologies, and learning about computer technologies. It also investigates 
relations between CIL and CT. 


1.9.1 Computer and information literacy 


The research questions concerned with CIL remain similar to those used in ICILS 2013. 
The questions are framed around variations in CIL and CT, the relationship of CIL to the 
characteristics of students, and the contexts in which CIL is developed. The research questions 
for ICILS 2018 also imply the analyses that could be used to address them, as well as a wide 
range of more specific research hypotheses. 


RQCIL1 What variations exist across countries, and within countries, in students’ CIL? 


RQCIL2 What aspects of schools and countries are related to students’ CIL? 


Following are some of the aspects of schools and education systems that could 
potentially be related to students’ CIL: 


a) General approaches and priorities accorded to computer and information 
literacy education at system and school level. 


b) School coordination and collaboration regarding the use of ICT in teaching. 


c) School and teaching practices regarding the use of technologies in students’ 
CIL. 


d) Teacher proficiency in, attitudes towards, and experience with using computers. 


e) ICT resources in schools. 


f) Teacher professional development. 


RQCIL3 What are the relationships between students’ levels of access to, familiarity with, 
and self-reported proficiency in using computers and their CIL? 
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RQCIL4 What aspects of students’ personal and social backgrounds (such as gender, and 
socioeconomic background) are related to students’ CIL? 


1.9.2 Computational thinking 


The proposed research questions relating to CT closely reflect those proposed for CIL, but 
exclude reference to changes from ICILS 2013 and include reference to the relationship between 
CT and CIL. Analyses will include data from those countries participating in the international 
option assessing students’ CT achievement. 


RQCT1. ~~ ~=What variations exist across countries and within countries, in students’ CT? 


RQ CT 2. What aspects of schools and countries are related to students’ CT? 


RQCT3 What are the relationships between students’ levels of access to, familiarity with, 
and self-reported proficiency in using computers and their CT? 


RQCT4 What aspects of students’ personal and social backgrounds (such as gender, and 
socioeconomic background) are related to students’ CT? 


RQCT5 What is the association between students’ CIL and CT? 


1.10 Participants and instruments 
1.10.1 Participants and sampling 


The ICILS target population comprises students in their eighth year of schooling. In most 
education systems, this is grade 8, provided that the average age of students in this grade is 13.5 
years or above. In education systems where the average age in grade 8 is below 13.5 years old, 
the adjacent upper grade (9) is defined as the ICILS target population. Schools with students 
enrolled in the target grade will be selected randomly proportional to size (PPS). Within each 
sampled school, 20 students are randomly selected from among all students enrolled in the 
target grade. 


The population for the ICILS teacher survey is defined as all teachers teaching regular school 
subjects at the target grade. It includes only those teachers who are teaching target grade 
students during the testing period and have been employed at school from the beginning of 
the school year. Up to 15 teachers are randomly selected from the teacher population at each 
sampled school. 


School-level data are provided by the principal and ICT coordinator from each sampled school. 
In addition, national centers will provide information about the national contexts for ClL and CT 
learning by drawing on relevant expertise in each country. 


1.10.2 Instruments 


The following instruments form part of ICILS. 


An international computer-based student test consisting of: 


e questions and tasks set in authentic contexts designed to measure students’ CIL? 


e questions and tasks set in authentic contexts designed to measure CT’. 


A student questionnaire consisting of a computer-based set of items measuring student 
background, access to, experience and use of, and familiarity with ICT at home and at school. The 
questionnaire also includes questions designed to gauge students’ attitudes toward using ICT. 


1 This is the core test instrument completed by students in all participating countries. 
2 This test instrument is completed only by students in countries participating in the international option of assessing CT. 
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A teacher questionnaire administered to selected teachers teaching any subject in the target 
grade. It gathers information about teacher background and their use of ICT. The questionnaire 
includes items that ask teachers to rate their confidence in using computers in their teaching, 
to state their actual use of computers, and to express their attitudes toward using computers in 
teaching and learning. 


A school principal questionnaire, administered to the principals of sampled schools and designed 
to capture school characteristics, the application of ICT in teaching and learning, and aspects of 
the management of ICT at school. 


An ICT coordinator questionnaire administered to ICT coordinators of sampled schools designed 
to capture information on resources and support for ICT at schools. 


A national contexts survey completed by ICILS national research centers drawing on relevant 
expertise in each country. The survey will gather information on the structure of the education 
system, the status of ClL-related education in the national curriculum and policies, initiatives 
and resourcing associated with ICT, and Cll-related education. The online questionnaire also 
includes questions related to extent in which CT learning is incorporated into the nationa 
educational policies (for instance, questions on the extent in which CT processes such as writing 
or evaluating code, programs or macros are included in the curriculum). The data obtained from 
this survey should provide a description of the contexts for ClL- and CT-related education in 
each country and assist with the interpretation of results from the student, school, and teacher 
questionnaires. 


Open Access This chapter is licensed under the terms of the Creative Commons Attribution-NonCommercial 4.0 
International License (http://creativecommons.org/licenses/by-nc/4.0/), which permits any noncommercial use, 
sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit 
to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes 
were made. 

The images or other third party material in this chapter are included in the chapter’s Creative Commons license, 
unless indicated otherwise in a credit line to the material. If material is not included in the chapter’s Creative 
Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you 
will need to obtain permission directly from the copyright holder. 


Check for 
updates 


CHAPTER 2 


Computer and information literacy 
framework 


2.1 Overview 


The ICILS CIL framework was initially developed for ICILS 2013. At the time, it was noted that 
there was a variety of terms relating to CIL in the research literature (see, for example Virkus 
2003) and that the development of context-specific constructs related to CIL had led to a 
proliferation of frequently overlapping and confusing definitions (Fraillon et al. 2013, p. 15). Since 
the development of ICILS 2013, the range of concepts associated with students’ use of digital 
technologies has further increased. For example, Siddiq et al. (2016, p. 60) listed nine different 
names for “concepts for describing what and how students acquire, use, adapt to and learn with 
technology,” ranging from “internet skills” through to “21st century skills’; these include the ICILS 
2013 conceptualization of CIL. This evolving breadth of conceptualizations of competencies 
associated with the use of digital technologies is, in part, a function of the range of local contexts 
(including local curriculum needs) as countries develop their own approaches to having students 
both learn how to use digital technologies and use digital technologies to support learning across 
other domains. 


When the CIL construct was first defined and described for use in ICILS 2013, it was necessary 
to locate ClL within the existing broad suite of constructs related to digital literacy and to clearly 
articulate the scope of the ClL construct. The reasoning underpinning this process is outlined in 
detail in the ICILS 2013 assessment framework (Fraillon et al. 2013). The following is a summary 
of the key decisions in this process with some reflection on their continued relevance for ICILS 
2018. 


The CIL construct was formulated during a period when there was a tension in the research 
literature between beliefs in the need to develop new constructs to describe and measure new 
skills being demonstrated with changes in technology and beliefs that the new skills should be 
assimilated into existing constructs. This tension is ongoing, and was described by Voogt and 
Roblin (2012, pp. 301-302) in their comparative analysis of international frameworks for 21st 
century skills as an “ongoing controversy on whether these terms are actually used to designate 
new competences, or rather to give greater emphasis to a specific set of long known competences 
that are considered as especially relevant to the knowledge society.” 


One of the conceptual challenges for ICILS 2013 was to decide whether the construct of CIL 
should address a new set of competencies or emphasize its connection to existing ones. The 
research team, in consultation with external experts, eventually opted for the second approach. 


Furthermore, the ICILS conceptualization of CIL needed to take into account two fundamental 
parameters of ICILS: 


e ICILS targets school-aged children (in their eighth year of schooling) 


e The assessment is completed using computers and focuses on computer use. 


The second of these parameters has necessitated the establishment of a working definition of 
computer for ICILS. In the final decades of the 20th century, the predominant concept of the 
computer associated with school-aged children was either as a desktop or laptop computer (but 
not a smartphone or tablet). These devices could be used for a range of educational purposes, 
including but not limited to: program development, use of productivity tools (such as word 
processing or spreadsheet tools), tuition applications, art and design tools, data collection, the 
conduct of simulations, and searching for information (such as from an encyclopedia). As the 
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skills; and digital literacy. What is common across the different conceptualizations of 21st century 
skills is that they comprise a broad range of skills that typically include a sub-set of skills common 
to ClLas defined in ICILS, but also extending well beyond the reach of what can be assessed ina 
study such as ICILS. ClL should be regarded as fitting neatly under the broader umbrella of 21st 
century skills, but with a focus on one of its dimensions. 


CILS was established to investigate the competencies associated with computer and information 
iteracies as the enabling components of digital competence and representing aspects of the 
broader suite of 21st century skills. The ICILS research team developed the CIL construct 
independently of specific curriculum goals; the construct focused on what Lampe et al. (2010, 
p. 62) referred to as technology-mediated educational priorities for middle-school students. 
These include finding and synthesizing relevant resources, connecting to people and networks, 
and knowing how to present and express oneself online in general and through online systems 
in particular. 


2.2 Defining computer and information literacy 


ICILS defined CIL for use in ICILS 2013 with reference to definitions and constructs associated 
with information literacy and computer literacy. Information literacy constructs developed first 
through the fields of librarianship and psychology (Bawden 2001; Church 1999; Homann 2003; 
Marcum 2002) and are acknowledged as having the following processes in common: identifying 
information needs, searching for and locating information, and evaluating the quality of 
information (Catts and Lau 2008; Livingstone et al. 2008; UNESCO 2003). Information literacy 
constructs evolved to include the ways in which the collected information can be transformed 
and used to communicate ideas (Catts and Lau 2008; Peters 2004). 


Early conceptualizations of computer literacy in education typically focused not on the logical 
reasoning of programming (or the syntax of programming languages) but rather on declarative 
and procedural knowledge about computer use, familiarity with computers (including their uses), 
and, in some cases, attitudes toward computers (Richter et al. 2000; Wilkinson 2006). Since 
that time, the concept of CT has gained increasing prominence across education systems, with 
perhaps some of the inevitable “definitional confusion” (Grover and Pea 2013, p. 38) that comes 
with new (or renewed) curriculum areas. While ICILS acknowledges that, conceptually, CT may 
be included in the broader concept of computer literacy (see, for example, diSessa 2000), ICILS 
2018 distinguishes the functional aspects of computer literacy that support the use of digital 
devices when managing digital information from the problem solving and algorithmic thinking 
features of computer literacy that are core to CT. The former are part of the CIL construct 
established in the first ICILS cycle in 2013, while the latter are part of the CT construct developed 
for ICILS 2018 as an international option. 


if 


The assumption that information is received, processed, and transmitted underlies the CIL 
construct. Computer literacy constructs typically attribute less importance than is attributed 
by information literacy constructs to the nature and constituent parts of the information 
processing that happens between reception and transmission. In essence, computer literacy 
focuses on a more direct path between reception and transmission than information literacy, 
which emphasizes the processual steps involved as information is evaluated and transformed 
Boekhorst 2003; Catts and Lau 2008). Over time, the originally distinct constructs of computer 
iteracy and information literacy converged into a broader dimension reflecting ICT literacy and 
digital literacy. 


Some scholars have emphasized the potential for information literacy and ICT skills to develop 
independently of each other. Catts and Lau (2008, p. 7) observed that “people can be information 
literate in the absence of ICT,’ and Rowlands et al. (2008, p. 295) commented that “the 
nformation literacy of young people, has not improved with the widening access to technology: 
n fact, their apparent facility with computers disguises some worrying problems.” The CIL skills 
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on in ICILS, however, address computer literacy skills in the context 
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society” (ETS 2002, p 2). 


e “ICT literacy is the ability of individuals to use ICT appropriately 
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participate effectively in society” (MCEETYA 2007, p. 14). 
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e Digital literacy is “.. 
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to access, manage and 


the ability to use digital technology, communications tools, and/or 


in a knowledge society” (Lemke 2003, p. 22). 


Common to these definitions is the assumption that individuals have th 


to use the technologies. All three definitions also list very similar sets of information literacy 
and communication processes. Each also maintains that individuals need to acquire these forms 


of literacy in order to participate and function effectively in society. Bi 


postulated six categories under which ICT literacy knowledge, skills, attitudes, values and ethics 
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unication technology; analyze 


e technical skills needed 
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hnology effectively; and 


ished in ICILS 2013 and maintained as the definition in ICILS 2018 is: 


Computer and information literacy refers to an individual's ability to use computers to 
investigate, create, and communicate in order to participate effectively at home, at school, in 
the workplace and in society. 


This definition relies on, and brings together, technical competence (computer literacy) and 
intellectual capacity (conventional literacies including information literacy) to achieve a highly 
context-dependent communicative purpose that presupposes and transcends its constituent 
elements. This view of CIL is congruent with Audunson and Nordlie’s (2003) conceptual model 
of information literacy and is most closely aligned with the ICT literacy construct evident in the 
first of the ICT and digital literacy definitions cited above. 
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2.3 Revising the structure of the computer and information 
literacy construct 

According to the ICILS 2013 assessment framework (Fraillon et al. 2013), ClL was described as 

comprising two strands each of which was specified in terms of a number of aspects. 

Strand 1 collecting and managing information comprises three aspects: 

e Aspect 1.1: Knowing about and understanding computer use 


e Aspect 1.2: Accessing and evaluating information 


e Aspect 1.3: Managing information. 


Strand 2 producing and exchanging information comprises four aspects: 
e Aspect 2.1: Transforming information 
e Aspect 2.2: Creating information 


e Aspect 2.3: Sharing information 


e Aspect 2.4: Using information safely and securely. 


The structure described above did not presuppose an analytic structure although, at the time 
of its development, the ICILS research team anticipated the possibility that Strands 1 and 2 
might lead to separate measurement dimensions. Analyses of the ICILS 2013 main survey data 
included an investigation of the dimensionality (for details regarding the analytic approach see 
Gebhardt and Schulz 2015) but the very high latent correlations between the two strands led to 
the decision to report CIL achievement as a single dimension. 


Following ICILS 2013, the project team together with ICILS 2013 national 
researchers evaluated the CIL construct with reference to its use throughout the 
full life-cycle of study. While the content of the construct was deemed to be appropriate, they 
identified potential improvements that could be made to the structure of the CIL construct. 
Firstly, positioning knowing about and understanding computer use (Aspect 1.1) within Strand 1 
the receptive strand) and using information safely and securely (Aspect 2.4) within Strand 2 (the 
productive strand) was problematic because it undermined the stated acknowledgement that 
each of these aspects was applicable across both the receptive and productive strands. At the 
time the ClL construct was specified this problem was acknowledged with the caveat that the 
aspects were included in the strands in which they were deemed to have the greatest applicability. 
However, on reflection, the ICILS research team decided that it would be better to remove any 
implication that either aspect was better associated with receptive or productive skills. 


Furthermore, in a time of increasing opportunity for young people to act as information creators 
and publishers, it became apparent that Aspect 2.3 (sharing information) should be afforded 
greater prominence in the structure of the CIL construct. 


In response to these concerns, and in consultation with ICILS national researchers, the project 
team established a revised structure for the ClL construct for ICILS 2018. It is important to note 
that the restructuring of the CIL construct was undertaken to better communicate the contents 
and emphases of the construct and to minimize overlap across the aspects of the construct. The 
change in structure neither means a change in ICILS assessment content nor presupposes a 
change to the analytic structure of the CIL construct. 
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The ClL construct includes the following elements: 


Structure of the ICILS 2018 computer and information 


e Strand: This refers to the overarching conceptual category for framing the skills and knowledge 
addressed by the CIL instruments. 


e Aspect: This refers to the specific content category within a strand. 


The construct comprises four strands, which each contain two aspects (summarized in Figure 2.1 


and described in detail in section 2.5). The aspects encompass the set of knowledge, skills, and 
understandings held in common by the range of definitions of ICT literacy and digital competency 
discussed previously. 


Figure 2.1: ICILS 2018 CIL framework 


Computer and information literacy refers to an individual’s ability to use computers to 
investigate, create and communicate in order to participate effectively at home, at school, 
in the workplace, and in the community. 
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Understanding computer use comprises two aspects: 
e Foundations of computer use 


e Computer use conventions. 


Aspect 1.1 Foundations of computer use 


Foundations of computer use includes the knowledge and understanding of the principles 
underlying the function of computers rather than the technical detail of exactly how they work. 
This knowledge and understanding underpins effective and efficient computer use including basic 
troubleshooting. At a declarative level, a person should know that computers use processors and 
memory to run programs, or that operating systems, word processors, games, and viruses are 
examples of programs. They should be able to demonstrate knowledge that computers can be 
connected and so can “communicate” with one another through networks, and that these can 
be local or global. They should understand that the internet is a form of computer network that 
is run through computers and that websites, blogs, wikis, and all forms of computer software 
are designed to meet specific purposes. They should further be aware that information (such as 
files) can be stored across a range of locations including locally on a device, on removable devices 
such as USB drives, SD cards and portable hard drives), and on local or remote networks (such 
as in cloud storage), and be aware that the range of storage locations are associated with a range 
of user benefits, risks and procedures. 


The following examples reflect tasks that provide evidence of an individual’s knowledge and 
understanding of the foundations of computer use: 


e identifying that computers require physical memory, and that this is finite but may be expanded 
e suggesting basic strategies to improve the performance of a computer that is running slowly 


e explaining why the content of a completed web-based form might be lost if a user navigates 
away from the page and then returns to the page 


e recognizing strategies to identify the part of a computer network that might be malfunctioning 
if a network connection has been lost. 


Aspect 1.2 Computer use conventions 


Computer use conventions include the knowledge and application of the software interface 
conventions that help computer users make sense of and operate software. This knowledge 
supports the efficient use of applications, including the use of devices or applications that are 
unfamiliar to the user. Accordingly, at the procedural level, a person may know how to execute 
basic, generic file and software functions, such as opening and saving files in given locations, 
resizing images, copying and pasting text, and identifying file types by their extensions, or 
modifying settings such as screen resolution or enabling accessibility options. The procedural 
knowledge included in Aspect 1.2 is thus limited to generic basic commands that are common 
across software (including operating system) environments. 


The following examples reflect tasks that provide evidence of an individual’s ability to apply 
computer use conventions: 


e Editing an image using an interface with icons and controls that follow software interface 
conventions 


e Clicking on a hyperlink to navigate to a web-page 


e Saving an existing file to a new location with a new name 


e Opening a file of a specified type 


e Adding users to a collaborative web-based workspace. 
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2.5.2 Strand 2: Gathering information 


Gathering information embraces the receptive and organizational elements of information 
processing and management. This strand comprises two aspects: 


e Accessing and evaluating information 


e Managing information. 


Aspect 2.1: Accessing and evaluating information 


Accessing and evaluating information refers to the investigative processes that enable a 
person to find, retrieve, and make judgments about the relevance, integrity, and usefulness of 
computer-based information. The proliferation of information sources that use the internet as 
a communication medium means that users are required to filter the vast array of information 
to which they gain access before they can make use of it. However, the process of filtering, in 
combination with the increasing intuitiveness of computer-based information search programs’, 
is producing an ever greater integration of the processes of accessing and evaluating information. 
For this reason, accessing and evaluating information are regarded as sufficiently integrated 
to warrant their inclusion as a single aspect, rather than as separate aspects, of the digital 
information dimension of the CIL construct. 


The importance of accessing and evaluating information is also a direct result of the increasing 
quantity and range of available unfiltered computer-based (and delivered) information. 
Computer-based information is not only increasing in volume, but also is constantly changing. 
While accessing and evaluating information are rooted in conventional literacies, the dynamic 
multimedia and multimodal nature of computer-based information means that the processes 
of accessing and evaluating that contribute to the CIL construct are different from those that 
relate only to conventional literacies. The dynamic context of computer-based information 
therefore necessitates the use of an amalgam of a range of skills (i.e., those typically associated 
with digital and media literacies) that differ from, and are broader than, the range employed with 
conventional literacies. 


The following examples reflect tasks that provide evidence of an individual's ability to access and 
evaluate computer-based information: 


e Selecting information from within a website or file list that is relevant to a particular topic 

e Describing and explaining the functions and parameters of different computer-based 
information search programs 

e Suggesting strategies for searching for information and/or adjusting the parameters of 
searches to target information better 

e Recognizing and explaining characteristics of computer-based information (such as hyperbole 
and unsubstantiated claims) that detract from its credibility 

e Recognizing that published information may serve other purposes than simply the sharing of 
information 


e Suggesting and implementing strategies to verify the veracity of information (such as cross- 
checking information from multiple sources). 


Aspect 2.2: Managing information 


Managing information refers to the capacity of individuals to work with computer-based 
information. This information can be in the form of files that can be stored and opened using 
applications for later use or data that can be organized within flles (such as data within fields ina 
database). The process of managing information includes ability to adopt and adapt information 
classification and organization schemes in order to arrange and store information so that it can 


3 These include search engines that tailor search results to individual searchers based on location, previous search 
behavior, and even internet-use behavior of “friends” in a social network. 
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be used or reused efficiently. Managing information can include applying procedures to make 


use of alternative file storage locations (such as local, local or remote network or cloud-based 
locations) to support user access and to back-up information to protect against loss. 


The following examples reflect tasks that provide evidence of an individual's ability to manage 
information: 


e Creating a file structure in a directory according to given parameters 


e Sorting or filtering information on an internet database 


e Explaining the use of tags when storing images in an image library 


e Recognizing the most efficient data structure for a given purpose within a simple database. 


2.5.3 St 


rand 3: Producing information 


This strand, which focuses on using computers as productive tools for thinking and creating, has 
two aspects: 


e Transforming information 


e Creating 


information. 


Aspect 3.1: Transforming information 


Transforming information refers to a person’s ability to use computers to change how information 


is presented so that it is clearer for specific audiences and pu 


involves using the formatting, graphics, and multimedia potentia 


communica 
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with a flow chart or diagram) 


e Creating 


a chart to represent a table of data 


e Transferring data (such as temperature or velocity data) from a 
ways that illustrate patterns of change 


e Creating 


Aspect 3.2: Creating information 


odifying, or creating images to supplement or replace text in a document (such as 


data logger and displaying it in 


a short animated sequence of images to illustrate a sequence of events. 


Creating information refers to a person’s ability to use computers to design and generate 


information products for specified purposes and aud 


iences. These original products may be 


entirely new or may build upon a given set of information to generate new understandings. 


Typically, the quality of information created relates to how the content is structured (whether or 
not the flow of ideas is logical and easy to understand) and the way in which layout and design 


features (such as images a 


nd formatting) are used to support understanding the resulting 


information product. Even though information design and layout design are executed together 
in an information product, they are typically conceptualized and 


of creating 


The followi 


information. 


e Using a simple graphics program to design a birthday card 


e Designin 


assessed as discrete elements 


ng examples reflect tasks that provide evidence of an individual's ability to create 
information: 


g and writing a presentation that explains the key elements of an historical event 


e Using a given set of information to make recommendations in a report that integrates text, 
data, and graphics. 
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2.5.4 Strand 4: Digital communication 


Digital communication focuses on the competencies associated with information sharing in 
social networking (and broader web-based information sharing space) together with the social, 
legal and ethical responsibilities associated with information sharing. This strand includes 
responsibilities associated with information production, as well as mechanisms for protection 
against improper use of information by others. This strand has two aspects: 


e Sharing information 


Aspect 4.1: Sharing information 


Using information safely and securely. 


Sharing information refers to a person’s understanding of how computers are and can be used, 
as well as his or her ability to use computers to communicate and exchange information with 
others. Sharing information focuses on a person’s knowledge and understanding of a range of 
computer-based communication platforms, such as email, wikis, blogs, instant messaging, sharing 
media, and social networking websites. Given the rapidly changing nature of this area, Aspect 
4.1 focuses on knowledge and understanding of information-based social conventions and, at 
the higher end of the achievement spectrum, the social impact of sharing information through 
computer-based communication media. 


The following examples reflect tasks that provide evidence of an individual’s ability to share 


information: 


Recognizing some key differences between computer-based communication media 


Using software to disseminate information (such as attaching a file to an email or adding or 
editing a social media post) 


Evaluating the appropriateness of information for a specified audience 


Evaluating the best communication platform for a particular communicative purpose 


Creating or modifying information products to suit a specified audience and purpose. 


Aspect 4.2: Using information responsibly and safely 


Using information responsibly and safely refers to a person’s understanding of the legal and 
ethical issues of computer-based communication from the perspectives of both the publisher and 
the consumer. Internet-based communication platforms increasingly are providing the facility 
for users to share information. With this facility comes the potential for misuse, particularly 
when dealing with personal information. Using information safely and securely also includes 
risk identification and prevention, as well as the parameters of appropriate conduct, including 
awareness of and prevention of cyberbullying. It furthermore focuses on the responsibility of 
users to maintain a certain level of technical computer security, such as using strong passwords, 
keeping virus software up to date, and not submitting private information to unknown publishers. 


The following examples reflect content and contexts relating to responsible and safe use: 


Identity theft 
Unauthorized access and impersonation 
Identity concealment 

hing 

alicious software distribution 


Automatic collection of internet usage data 


Social network posts 


Provision and use of personal information 


Attribution and copyright. 
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The following examples reflect tasks that provide evidence of an individual’s ability to use 
information safely and securely: 


e Identifying the characteristics that influence the strength of passwords 

e Explaining the consequence(s) of making personal information publicly available 
e Describing protocols for appropriate behavior on a social media site 

e Suggesting ways to protect private information 


e Understanding how internet advertising targets users 


e Explaining the techniques used in a phishing email scam. 
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CHAPTER 3 


Computational thinking framework 


3.1 Overview 


The ICILS CIL construct was established and measured in response to the pervasive belief in 
the value of ClL-related competencies for participation in the 21st century. At the same time 
as ICILS 2013 was being developed, there was the beginning of a resurgence of interest from 
researchers, educators, and policymakers in the importance of CT in education (Voogt et al. 
2015). The inclusion of CT as an international option in ICILS 2018 was, in part, a response to 
a growing belief in the importance of computer science and computational thinking in schooling 
and efforts across countries to expand students’ access to these areas of learning (Yadav et al. 
2018). 


While CT has been recognized “since the beginning of the computing field in the 1940s” (Denning 
2017, p 34), many researchers refer to the work of Papert in the late 20th century (Papert 1980, 
1991; Shute et al. 2017; Voogt et al. 2015) as atouchstone for CT research. More recently, Wing's 
(2006) article on CT has been regarded by researchers as the catalyst, or at least as a common 
point of reference, for the re-emergence of interest in CT (see, for example, Barr and Stephenson 
2011; Bower et al. 2017; Grover and Pea 2013; Shute et al. 2017; Voogt et al. 2015). In this 
article, Wing characterized CT as “a universally applicable attitude and skill set everyone, not just 
computer scientists, would be eager to learn and use” (Wing 2006, p. 33). However, despite the 
high level of interest in CT and the rapid increase in curriculum and educational resources, along 
with research concerned with CT, there has been confusion about its definition (Denning 2017; 
Grover and Pea 2013; Selby and Woollard 2013). This confusion is partly attributable the broad 
range of perspectives on CT that abound. For example, the National Academic Press reported 
on a 2009 workshop on the nature of CT that cited the following perspectives on CT (National 
Research Council 2010, pp. 11-12): 


e CT is “closely related to, if not the same as...procedural thinking...that includes developing, 
testing, and debugging procedures” 


e CT is about “expanding human mental capacities through abstract tools that help manage 
complexity and allow for automation of tasks” 

e CT is primarily about processes and is a subset of computer science 

e CT is “the use of computation-related symbol systems (semiotic systems) to articulate 
explicit knowledge and to objectify tacit knowledge to manifest such knowledge in concrete 
computational forms” 


( 


e CT is about “rigorous analyses and procedures for accomplishing a defined task” 


e CT “is a bridge between science and engineering—a meta-science about studying ways or 
methods of thinking that are applicable to different disciplines” 


e CTis “what humans do as they approach the world [that is, their framing, paradigm, philosophy, 
or language], considering processes, manipulating digital representations (and [meta] models), 
and hence all humans engage in computational thinking to some extent already in their daily 
lives” 


e CT “plays a role in the manipulation of software in support of problem solving” 


e “What makes computational thinking especially relevant is that computers can execute our 


»” 


‘computational thoughts’. 
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The range of different perspectives listed above exemplify some of the tensions that exist in 
approaches to CT. These tensions are associated with identifying where CT should be located on 
a spectrum of capabilities that, at one end, are characterized by algorithmic procedural thinking 


associated with co 


mputer programmin 


g and 


, at the other end, are described by a broader suite 


of problem solving capabilities and dispositions (see, for example, Barr et al. 2011; Barr and 
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ICILS assessment parameters. In this case the assessment of CT must be: 


e Appili 
Appli 


Com 


math 
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cable to students in their eighth year of schooling 


plementary to the ICILS assessment of CIL 


cable across a broad range of country and curriculum contexts 


I, as for CIL, must be considered in the context of the 


Minimally overlapping with assessment content in other curriculum areas (such as in 
ematics or science). 


ese parameters in mind, the conceptualization of CT in ICILS is that CT combines the 


competencies associated with (a) framing solutions to real-world problems in a way that these 
solutions could be executed by computers; and then (b) implementing and testing solutions using 
the procedural algorithmic reasoning that underpins programming. 


3.2 


Ina review of CT literature, Selby and Wool 


constituen 
abstraction 


Defining computational thinking 
ard (2013) identified three consistently shared 
t components of CT: (a) a thought process (a way of thinking about computing); (b) 
(describing the co 
and (c) decomposition (breakin 


mmon underlying properties a 
g acomplex problem into well- 


720) suggested that many definitions of CT focus on the “skil 


solve complex problems with 


the help of computing.” 


nd functionality of a set of 
defined parts). Voogt et al. 
s, habits and dispositions n 


entities); 
(2015, p. 
eeded to 


Following is a selection of definitions and descriptions of CT that have been used to inform the 

development of the definition of CT established for use in ICILS. 

(1) “Computational thinking is the thought processes involved in formulating problems and 
their solutions so that the solutions are represented in a form that can be effectively carried 
out by an information-processing agent” (Wing 2011, as cited by Grover and Pea 2013, p. 
39). 

(2) “We consider computational thinking to be thought processes involved in formulating 
problems so their solutions can be represented as computational steps and algorithms” 
(Aho 2012, p. 832). 

(3) “It [computational thinking] is a cognitive or thought process that reflects: 

e the ability to think in abstractions, 

e the ability to think in terms of decomposition, 

e the ability to think algorithmically, 

e the ability to think in terms of evaluations, and 

e the ability to think in generalizations” (Selby and Woollard 2013, p. 5). 

(4) “Computational thinking describes the processes and approaches we draw onwhen thinking 
about how a computer can help us to solve complex problems and create systems’ (Digital 


Technologies Hub 2018). 
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(5) “Computational thinking is the process of recognizing aspects of computation in the world 
that surrounds us, and applying tools and techniques from Computer Science to understand 
and reason about both natural and artificial systems and processes” (Royal Society 2012, 
p. 29). 


(6) “Computational thinking is a problem-solving process that includes: 


e Formulating problems in away that enables us to use a computer and other tools to help 
solve them 


e Logically organizing and analyzing data 


e Representing data through abstractions, such as models and simulations 


e Automating solutions through algorithmic thinking (a series of ordered steps) 


e |dentifying, analyzing, and implementing possible solutions with the goal of achieving the 
most efficient and effective combination of steps and resources 


e Generalizing and transferring this problem-solving process to a wide variety of problems” 
(Barr et al. 2011, p. 21). 
(7) “Computational thinking is a term often used to describe the ability to think with the 
computer-as-tool” (Berland and Wilensky 2015, p. 630). 


Common to these definitions of CT is the idea that CT is regarded as a form of problem solving 
in which the problems and solutions are conceptualized so that algorithmic, procedural (step- 
by-step) solutions can be established and implemented using a computer. These characteristics 
are consistent with the ICILS conceptualization of CT as focusing on problem solving to 
generate computer-based solutions. While it can reasonably be argued that the core of this 
conceptualization of CT may be applied to other learning domains, the ICILS test of CT does not 
include measurement of cross-domain applications of CT. 


The definition of CT established within the context of ICILS is: 


Computational thinking refers to an individual's ability to recognize aspects of real-world 
problems which are appropriate for computational formulation and to evaluate and develop 
algorithmic solutions to those problems so that the solutions could be operationalized with 
a computer. 


3.3 Structure of the ICILS 2018 computational thinking 
construct 
The CT construct includes the following elements: 


e Strand: This refers to the overarching conceptual category for framing the skills and knowledge 
addressed by the CT instruments. 


e Aspect: This refers to the specific content category within a strand. 


The CT construct comprises two strands. One strand contains three aspects and the other 
comprises two aspects (summarized in Figure 3.1 and described in detail in section 3.4). The 
aspects encompass the set of knowledge, skills, and understandings held in common across the 
range of definitions of CT as discussed previously. 
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Figure 3.1: ICILS 2018 CT framework 


Computational thinking refers to an individual’s ability to recognize aspects of real-world 
problems which are appropriate for computational formulation and to evaluate and develop 
algorithmic solutions to those problems so that the solutions could be operationalized with 
a computer. 


Strand 1 Strand 2 
Conceptualizing problems Operationalizing solutions 
Aspect 1.1 Aspect 2.1 
Knowing about and understanding digital Planning and evaluating solutions 
t 
arene Aspect 2.2 
Aspect 1.2 Developing algorithms, programs and 
Formulating and analyzing problems interfaces 
Aspect 1.3 


Collecting and representing relevant data 


The structure shown above does not presuppose a sub-dimensional structure of the CT construct. 
The primary purpose of describing CT using this structure is to organize the CT content in a way 
that allows readers to clearly see the different related aspects of CT and to support the auditing 
of the CT instruments against the full breadth of content in the CT construct. We hypothesize 
that CT will form a single measurement dimension. However, analyses of the dimensionality 
of the ICILS 2018 CT data will be used to determine whether CT is reported as a single or as 
multiple dimensions. 


3.4 Strands and aspects of computational thinking 


3.4.1 Strand 1: Conceptualizing problems 


Conceptualizing problems acknowledges that before solutions can be developed, problems must 
first be understood and framed in a way that allows algorithmic or systems thinking to assist in 
the process of developing solutions. This strand comprises three aspects: 


e Knowing about and understanding digital systems 


e Formulating and analyzing problems 


e Collecting and representing relevant data. 


Aspect 1.1: Knowing about and understanding digital systems 


Knowing about and understanding digital systems refers to a person’s ability to identify and 
describe the properties of systems by observing the interaction of the components within a 
system. 


Systems thinking is used when individuals conceptualize the use of computers to solve real- 
world problems, which is fundamental to computational thinking. 


At a declarative level a person can describe rules and constraints that govern a sequence of 
actions and events, or they are able to provide a prediction for why a procedure is not working 
correctly by observing the conditions of the error. For example, imagine a student was required 
to design a game. The student would first need to specify the initial state of the game, the winning 
condition of the game and the parameters of the permissible actions, and sequence of actions 
within the game. 
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At a procedural level, a person can monitor a system in operation, make use of tools that help to 
describe a system (such as tree diagrams or flow charts), and observe and describe outcomes 


if 


O 


and the outcomes of varying th 


a processes operating within a system. These procedural skills are based on a conceptual 
understanding of fundamental operations such as iteration, 


ooping, and conditional branching, 
e sequence in which they are executed (control flow). An 


understanding of these operations can enhance a person's understanding of both the digital 
world and the physical world; and it can therefore assist in solving problems. With reference 
to the example of a student designing a game, at the procedural level the student might initiate 
and adjudicate the game play. The student would need to monitor the players’ actions and the 


consequent outcomes according to the specified rules and conditions of the game. In doing this, 
the student may observe problems with the game, such as unresolvable or ambiguous situations 


and be able to modify the game pa 


rameters accordingly. It is not always necessary that the game 


be created as acomputer application, as digital systems thinking can also be applied to non-digital 


systems. In the context of ICILS, di 
of a physical system (such as filling 


gital systems thinking could be applied to describe the actions 
a glass with water from a tap) in such a way that these actions 


could later be controlled by a computer program. 


The following examples reflect tasks that provide evidence of an individual’s ability to know 
about and understand digital systems: 


e Exploring a system to describe 


rules about its behavior 


e Operating a system to produce relevant data for analysis 


e |dentifying opportunities for efficiency and automation 


e Explaining why simulations help to solve problems. 


Aspect 1.2: Formulating and analyzing problems 


specifying and systematizing the c 
be developed (possibly with the ai 


Formulating problems entails the decomposition of a problem into smaller manageable parts and 


haracteristics of the task so that a computational solution can 
d of a computer or other digital device). Analyzing consists of 


problems to establish a conceptua 
problem into a set of 


making connections between the properties of, and solutions to, previously experienced and new 


framework to underpin the process of breaking down a large 


The following examples reflect tas 
and analyze problems: 


e Breaking down a complex task 


smaller, more manageable parts. 


s that provide evidence of an individual’s ability to formulate 


into smaller, more manageable parts 


e Creating a self-contained sub-task that could potentially be applied more than once 


e Exploring the connection between the whole and the parts. 


Aspect 1.3: Collecting and representing relevant data 


In order to make effective judgements about problem solving within systems it is necessary 


to collect and make sense of data 


from the system. The process of collecting and representing 


data effectively is underpinned by knowledge and understanding of the characteristics of the 


data and of the mechanisms availa 
This could involve creating or usin 


ble to collect, organize, and represent these data for analysis. 
g asimulation of a complex system to produce data that may 


show patterns or characteristics of behavior that are otherwise not clear when viewed from an 


abstract system level. 


The following examples reflect tasks that provide evidence of an individual's ability to collect and 


represent data: 


e |dentifying an abstracted representation of map directions 


e Using a route simulation tool to store data 


e Displaying data to help draw conclusions and inform planning 


e Using simulation tool to collect 


data and evaluate outcomes. 
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3.4.2 Strand 2: Operationalizing solutions 


Operationalizing solutions comprises the processes associated with creating, implementing and 


evaluating computer-based system responses 


to real-world problems. It includes the iterative 


processes of planning for, implementing, testing, and evaluating algorithmic solutions (as the 
potential bases for programming) to real-world problems. The strand includes an understanding 
of the needs of users and their likely interaction with the system under development. The strand 


comprises two aspects: 


e Planning and evaluating solutions 


e Developing algorithms, programs and interfaces. 


Aspect 2.1: Planning and evaluating solutions 


Planning solutions refers to the process of es 


Evaluating solutions refers to the ability to 


tablishing the parameters of a system, including 


the development of functional specifications or requirements relating to the needs of users and 
desired outcomes and with a view to designing and implementing the key features of a solution. 


make critical judgements about the quality of 


computational artefacts (such as algorithms, code, programs, user interface designs, or systems) 
against criteria based on a given model of standards and efficiency. These two processes are 


combined ina single aspect because they are iteratively connected to the process of developing 


algorithms and programs. While the process of developing algorithms may begin with planning 


and end with evaluation, throughout the proce 


ss there is a constant iteration between planning, 


implementation, evaluation, and revised planning (or resolution). Typically, there is a broad 


array of potential solutions to any given probl 
to plan and evaluate solutions from a range of 
disadvantages, and effects on stakeholders of 


em and, consequently, it is important to be able 
perspectives, and to understand the advantages, 
alternative solutions. 


The following examples reflect tasks that provi 


evaluate computational solutions: 


de evidence of an individual’s ability to plan and 


e |dentifying the starting point for an algorithmic solution to a problem by reflecting on solutions 


to similar problems 


e Designing components of a solution taking i 
needs of users 


e Testing a solution method against a known 


nto account the limitations of the system and the 


outcome and adjusting it as necessary 


e Comparing the relative advantages and disadvantages of a solution against alternative 


solutions 


e Locating a faulty step in an algorithm 


e Describing solutions and explaining why they are the best solution among many 


e Implementing and managing strategies to test the efficacy of a solution (such as user testing). 


Aspect 2.2: Developing algorithms, programs and interfaces 


ICILS 2018 does not presuppose that students are familiar with the syntax and features of 


any particular coding language. This aspect f 


implementing an algorithm (systematically des 


ocuses on the logical reasoning that underpins 


the development of algorithms (and code) to solve problems. It can involve developing or 


cribing the steps or rules required to accomplish 


a task) and also automating the algorithm, typically using computer code in a way that can be 
implemented without the need for students to learn the syntax or features of a specific coding 


language. Creating an interface relates to the intersection between users and the system. 


This may relate to development of the user 


interface elements in an application including 


implementation of specifications for dynamic interfaces that respond to user input. 
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The following examples reflect tasks that provide evidence of an individual’s ability to develop 
algorithms, programs, and interfaces include the following: 


e Modifying an existing algorithm for a new purpose 


dapting visual directions into instructions for a computer 


reating visual representations of instructions for a computer 


sing a new statement in a simple algorithm 


A 
C 
e Creating a simple algorithm 
U 
C 


reating an algorithm that combines simple command statements with a repeat or conditional 


statement 


e Correcting a specified step in an algorithm. 


Open Access This chapter is licensed under the terms of the Creative Commons Attribution-NonCommercial 4.0 
International License (http://creativecommons.org/licenses/by-nc/4.0/), which permits any noncommercial use, 
sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit 
to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes 
were made. 

The images or other third party material in this chapter are included in the chapter’s Creative Commons license, 
unless indicated otherwise in a credit line to the material. If material is not included in the chapter’s Creative 
Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you 
will need to obtain permission directly from the copyright holder. 


Check for 
updates 


CHAPTER 4 


Contextual framework 


4.1 Overview 


This chapter describes the contextual information collected during ICILS 2018 in order to aid 
understanding of variation in the primary outcome achievement measures of the study: students’ 
computer and information literacy (CIL) and computational thinking (CT). Throughout this 
chapter, the abbreviation CIL/CT has been used where each of ClL and CT may be considered as 
an outcome measure potentially influenced by a given set of contextual information. We provide 
a Classification of contextual factors that accords with the multilevel structure inherent in the 
process of student CIL/CT learning, and consider the relationship of these factors to the learning 
process (antecedents or processes). We also list the different kinds of variables that will be 
collected via the different ICILS 2018 contextual instruments and briefly outline prior findings 
from educational research in order to explain why these variables are included in ICILS 2018. 


4.2 Classification of contextual factors 


When studying student outcomes related to CIL/CT, it is important to set these in the context 
of the different factors influencing them. Students acquire competencies in this area through 
a variety of activities and experiences at the different levels of their education and through 
different processes in school and out of school. It is also likely, as Ainley et al. (2009) argued, that 
students’ out-of-school experiences of using ICT influence their learning approaches in school. 
Contextual variables can also be classified according to their measurement characteristics, 
namely, factual (e.g., age), attitudinal (e.g., enjoyment of computer use), and behavioral (e.g., 
frequency of computer use). 


Different conceptual frameworks for analyzing educational outcomes frequently point out the 
multilevel structure inherent in the processes that influence student learning (see, for example, 
Gerick et al. 2017; Hatlevik et al. 2015; Schulz et al. 2016; Vanderlinde et al. 2014). The learning 
of individual students is set in the overlapping contexts of school learning and out-of-school 
earning, both of which are embedded in the context of the wider community that comprises local, 
national, supranational, and international contexts. As for ICILS 2013, the contextual framework 
of ICILS distinguishes the following levels: 


e Wider community: This level describes the wider context in which CIL/CT learning takes place. 
t comprises local community contexts (e.g., remoteness and access to internet facilities), as 
well as characteristics of the education system and country. Furthermore, it encompasses the 
obal context, a factor widely enhanced by access to the internet. 


8 
e Schools and classrooms: This context encompasses all school-related factors. Given the cross- 
urricular nature of CIL/CT learning, it is not useful to distinguish between classroom level 
nd school level. 


C 

a 
e Home environment: This context relates to the student's background characteristics, especially 
nterms of the learning processes associated with family, home, and other immediate out-of- 
school contexts. 


e The individual: This context includes the characteristics of the student, the processes of 
learning, and the student’s level of CIL/CT. 
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The status of contextual factors within the learning process is also important. Factors can be 
classified either as antecedents or processes: 


e Antecedents are exogenous factors that condition the ways in which CIL/CT learning takes 
place. They are contextual factors that are not directly influenced by learning-process 
variables or outcomes. It is important to recognize that antecedent variables are level specific 
and may be influenced by antecedents and processes found at higher levels, for example, the 
extent to which schools’ ICT resources are likely to be influenced by ICT education policies at 
the level of the education system. 


e Processes are those factors that directly influence CIL/CT learning. They are constrained by 
antecedent factors and factors found at higher levels. This category contains variables such as 
opportunities for CIL/CT learning during class, teacher attitudes toward using ICT for study 
tasks, and students’ use of computers at home. 


Both antecedents and processes need to be taken into account when explaining variation in CIL 
learning outcomes. Whereas antecedent factors shape and constrain the development of CIL, 
process factors can be influenced by the level of (existing) ClLlearning. For example, the level and 
scope of classroom exercises using ICT generally depend on the existing ClL-related proficiency 
of the students. 


In the basic classification of antecedent and process-related contextual factors in their 
relationship with CIL/CT outcomes located at the different levels, each type of factor at each 
level is accompanied by examples of variables that have the potential to influence learning 
processes and outcomes (Figure 4.1). It is important to note that there is a reciprocal association 
between learning processes and learning outcomes while there is a unidirectional influence 
between antecedents and processes. 


Figure 4.1: Contexts for ICILS 2018 CIL/CT learning outcomes 


Antecedents Processes Outcomes 

Wider community Wider community 
Educational system Educational policies 
Availability of ICT Internet sources 

School/classroom: School/classroom: 

Characteristics ICT use for teaching/ 

Stated ICT curriculum learning 

ICT resources CIL/CT instruction 


Computer and 


I I I i] 

! Student: ! ! Student: I : ae 

F = a , ; ; Pn ae i «3 information literacy 
1 aracteristics 1 1 BP I Computational thinking 
I I I i] 

I I I i] 

I : I I ; 1 

1 Home environment: 1 1 Homeenvironment: 

! Family background I ' ICT use at home ! 

I r I 1 

, and ICT resources i ' ' 


Notes: The double arrow between process-related factors and outcomes emphasizes the possibility of a reciprocal 
association between learning processes and learning outcomes. The single-headed arrow between antecedents and 
processes indicates the assumption within the ICILS contextual framework of a unidirectional influence between these two 
types of contextual factors. 
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Reference to this general conceptual framework enables potential contextual factors to be 
located on a two-by-four grid, where antecedents and processes constitute the columns and 
the four levels the rows (Table 4.1 shows examples of the contextual variables collected by the 
ICILS 2018 instruments). The student questionnaire will primarily collect data on contextual 
factors pertaining to the level of the individual student and his or her home context. The teacher, 
school principal, and ICT coordinator questionnaires are designed to locate contextual factors 
associated with the school/classroom level, while the national contexts survey and other available 
sources (e.g., published statistics) will gather contextual data at the level of the wider community. 


Table 4.1: Mapping of variables to contextual framework (examples) 


Level of ... Antecedents Processes 

Wider community NCS & other sources: NCS & other sources: 
Structure of education Role of ICT in curriculum 
Accessibilty of ICT 

School/classroom PrQ, ICQ, & TQ: PrQ, ICQ, TQ and StQ: 
School characteristics ICT use in teaching and learning 
ICT resources CIL/CT instruction 

Student StQ: StQ: 
Gender ICT activities 
Age Use of ICT 

CIL/CT 

Home environment StQ: StQ: 
Parent socioeconomic status Learning about ICT at home 
ICT resources 


Notes: NCS = national contexts survey; PrQ = principal questionnaire; ICQ = ICT coordinator questionnaire; TQ = teacher 
questionnaire; StQ = student questionnaire. 


4.3 Contextual levels and variables 


4.3.1 The wider community context 


Levels within the wider community context all have the potential to affect student learning at 
school or at home. Conceptually, this context has several levels: 


e local communities, where remoteness and lack of stable and fast internet connections may 
affect conditions for ICT use 


e Regional and national contexts, where communication infrastructure, educational structures, 
curricula, and general economic/social factors may be of importance 


e Supranational or even international contexts, where a long-term perspective brings in, for 
example, factors such as the general advance of ICT on a worldwide scale. 


The most important factors potentially explaining variation in CIL/CT are located at the national 
evel (or subnational level in those instances of sub-regions participating in the study). There is 
evidence of broad differences across countries in terms of access to digital technology across 
Europe, as well as more broadly across the world (Fraillon et al. 2014; Pew Research Center 
2015; World Bank 2016). 


£ 


5D 


ormation relating to the contexts of educational systems will primarily be sourced from the 
CILS 2018 national contexts survey, and supplemented by information from external databases 
and other published sources. Typically, these published sources provide information about 
antecedent country-context variables, while the national contexts survey will deliver data on 
antecedent and process variables at the education-system level. 
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More specifically, the national contexts survey is designed to collect systemic data on the 
following: 


e Structure and makeup of the educational system (with specific focus on the target grade) 


e Educational policy and practice in CIL/CT education (including curriculum approaches to CIL 
and CT) 


e Policies and practices for developing the CIL/CT expertise of teachers 


e Current debates on and reforms to the implementation of digital technology in schools 
including approaches to the assessment of CIL/CT and the provision of ICT resources in 
schools) 


oe 


e Information about ICT-based learning and administrative management system 


Antecedent variables at the level of the wider community 


International comparative research shows relatively strong associations between the general 
socioeconomic development of countries and student learning outcomes. ICILS 2018 will 
therefore select national (and where appropriate possible subnational) indicators related to 
general human development status as regularly reported by the United Nations Development 
Programme (UNDP 2016). Examples of these indicators are gross domestic product per person, 
access to education, and health statistics. 


Given ICILS’ focus on students’ CIL/CT, it is important to take into account the general availability 
of and infrastructure for ICT. To this end, ICILS 2018 will collect, with the aim of describing the 
general |CT-related resources at the national level, information relating to variables such as the 
proportion of the population with access to the internet. 


One example of a published source of data regarding national contexts is the ICT Development 
Index (IDI), developed by the International Telecommunications Union (ITU 2017). The IDI 
combines 11 indicators into a single measure that can be used as an index of ICT development for 
154 countries or used as separate indicators. Another index is the Networked Readiness Index 
(see, for example, Dutta and Mia 2011). 


Data from a range of international surveys show that the provision of ICT resources in schools 
varies widely across countries (see, for example, Anderson and Ainley 2010; Fraillon et al. 2014; 
Pelgrum and Doornekamp 2009). In order to obtain information related to the general policies 
regarding the ICT resourcing of schools, the ICILS 2018 national contexts survey will collect data 
about approaches to the provision of school-based ICT infrastructure, hardware, and software, 
as well as policy expectations regarding these provisions. 


These system-level data will be complemented by school-level information from the ICT 
coordinator questionnaire, which will collect information about indicators such as the number of 
computers per student, software licensing arrangements, and the availability of digital curriculum 
resources. 


The national contexts survey will also gather data about a range of other characteristics of the 
education systems participating in ICILS 2018. System-level variables related to this aspect 
include length of schooling, age-grade profiles, and structure of school education (e.g., study 
programs, public/private management), as well as the degree of autonomy of educational 
providers. 


Process-related variables 


The process-related variables on CIL/CT-related education policy that will be collected by the 
ICILS 2018 national contexts survey include: 


e The definition of and the priority that each country gives to CIL education in its educational 
policy and provision 


e Reforms in the use of ICT in education 


CONTEXTUAL FRAMEWORK 


Th 
Su 
Th 


fo 


influence of di 


S 


Because the initial ICIL 
as outcomes of earlier 
relating to students’ le 


and d 
Ed 


a 


untr 


e in 
p 
b 
p 
wan 
G) 


In order to build on what is a 
the inclusion of CIL/CT education (as a separate subj 
in the formal curriculu 
ll also capture the nomenc 


as a Ci 


different study programs. It wi 


ucation Study (S 
Primary and Secon 
periences with Tech 


e emphasis on CIL/CT learning in the curriculum 


pport by education authorities for teacher professio 


S§ 2013 contextual framework refere 


arnin 


g contexts and learning proces 


erent institutions or groups on decisions relating to those goals an 


arge-scale surveys of ICT in education, ICILS 2018 also considers t 


nal learning in CIL/CT education 


daims. 


nces policies and practices developed 


he data 


tabases from t 


TES) 
dary 


2011). 


ies (Bakia et al. 


formation from 


oss-curricular approach 


subjects and whether they are 


be specifi 
education. 


An 


applicable) data on |CT-related requirements for becoming a teacher. The survey also see 
information on the extent to which ICT-related educatio 


educ 
ICT 
learn 


other i 
expertise 
Scherer a 
opportuni 
approaches to te 


mportant process-re 


nd Siddi 
ties tod 


these st 
ementation of CIL/CT educ 
ject within the curriculum, w 
icitness with which countries describe their CIL/CT 
t from them also vary across education systems. S 
L education and its expected learning outcomes; 
“implicit” curriculum that weaves through the curricu 


c questions regardin 


hese studies. These studies include | 
(Plomp et al. 2009), the 
Educ 
nology in 


ation (Pelgrum and D 
Education survey, whi 


udies shows that co 


hereas others include 


European Com 


untries take di 
ation in their curricula. Some educa 


EA’s Seco 


oor 


ses that were included in the reports 
nd Information Tech 
mission’s Indicators 
ekamp 2009), and the Intern 


nology 
of ICT 
ational 


ch covered po 


it by integrating it into 
| curricula and the lear 
ome 


um documents for ot 


ready known, the nati 


onal contexts survey 


ect, integrated into d 
m at different stages 


compulsory or optiona 


others 


her lear! 


will gat 
ifferent 
of scho 


ature for CIL/CT-related 
in each program of study. Th 
g the target grade in terms of curriculum emphasis on CIL/CT 


icies and experiences in 21 


fferent approaches to the 
tion systems inclu 


de itasa 


ubjects. The 


ning outcomes they 
have very explicit Curricu 
others describe CIL/CT educati 


la regarding 
on as an 
ning areas. 
her data on 
subjects, or 
oling and in 
curriculum 
ere will also 


in |CT-related teachi 


Over t 
expa 


over th 


he past few decades, 


e most recent decade 


countr 
into sc 


how education systems assessed | 
to assess other disciplines. The IC 


hool education, includi 


owever, results from 


ng th 
learning objects (Fraillon et al. 20 


in education, on the providers of these activities, and 
ing about developments in CIL/CT education. 


ated variable at the system level is the development of teacher 
ng and learning (Charalambos and Glass 2007; Law et al. 2008; 
q 2015). Teacher education programs often provide aspiring teachers with 


evelop ICT-related competencies. To aid assessment of the variety of different 


acher education in the field, the national contexts survey gathers (where 


s out 


nis part of preservice or initial te 
ation, on the availability of in-service or continuing professional development for the use 


acher 
of 


on expectatio 


many education systems h 
nsion in the use of digital technology in education‘. A key feature of most national 
is that they aspire to u 
teaching, and also to develop capabilities useful wi 
improve existing practice. H 


Se 


14). Similarly, IC 
CT-related learni 


ave undertake 


ICT to transform 


ns for teachers’ on 


n reforms involving t 


patterns of learnin 


thin modern economies, rather than simp 
ICILS 2013 suggested that participatin 
ies differ in the extent to which they have introduced, or are introducing, digital technology 
e development of curriculum resources in the form of digit: 
LS 2013 found there was also variation 
ng outcomes and in whether they used ICT 


going 


ne 
Ns 


pla 
ga 


d 
y 
§ 


al 
in 


LS 2018 national contexts survey will therefore gather data 


about the priorities accorded to these aspects and the nature of corresponding debates about 
related policies. 


4 Two recent examples include the Slovak 


Republic's program for the “digitalization of the system of education” (Slovak 


Republic Ministry of Education 2013) and Lithuania's “Strategy on ICT integration into general and vocational education” 
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4.3.2 School/classroom context 


Any study of students’ acquisition of CIL/CT must acknowledge the key role that school and 
classroom contexts play in that acquisition. Use of ICT is increasingly becoming standard practice 
in education and is therefore an important part of preparing young people for participation 
in modern society. Factors associated with the school and classroom context will be collected 
through the teacher, school principal, and ICT coordinator questionnaires. In addition, the 
student questionnaire includes some questions gauging student perceptions about classroom 
practices related to ICT. Even though ICILS 2018 will not attempt to investigate the relationship 
between ICT use in schools or classrooms and achievement in academic learning areas such as 
language, mathematics, or science, it is of interest to note the evidence of a positive impact of ICT 
use on classroom achievement in a meta-analysis conducted by Tamin et al. (2011). 


Antecedent variables at the school/classroom level 


In line with the need to consider basic school characteristics in the analysis of variations in CIL/ 
CT, the school principal questionnaire will collect information on student enrollment, teachers, 
the range of grades, and the location of each participating school. Itwillalso collect data on school 
management (public or private). Because, as noted earlier, |CT-related resources at school can 
be regarded as an important contextual factor to consider when studying students’ CIL/CT, the 
school principal questionnaire will furthermore ask who, in the school, assumes responsibilities 
for the acquisition of ICT resources. 


School-level factors related to ICT resourcing and priorities are known to influence both the way 
in which teachers use ICT for teaching and learning, and students’ ICT-related learning (Fraillon et 
al. 2014; Gerick et al. 2017). The ICILS questionnaire for each school’s ICT coordinator includes 
questions on the availability of school-owned computing devices at school, their location within 
the school, how many students have access to them, and the number of years the school has been 
using ICT. The instrument will also collect data on the support the school provides for ICT use in 
teaching and learning in terms of personnel and technology or software resources. It additionally 
includes a question measuring the coordinator’s perceptions of the adequacy of the ICT on hand 
for learning and teaching at school. Analysis of this type of information will support evaluation of 
the premise that students in those schools with the highest levels of digital resourcing will have 
greater experience of and access to the use of CIL/CT, and consequently develop higher levels 
of CIL/CT. 


With regard to school-level antecedent variables potentially influencing the development of 
students’ CT skills, school |CT-coordinators are asked whether their school offers a stand-alone 
computing subject for the target grade, and, if so, to which extent this subject emphasizes a range 
of activities directly related to CT (e.g., activities such as developing algorithms, or debugging 
computer code). 


The background and experiences of teaching staff potentially influence the acquisition of 
students’ CIL/CT. Teachers’ sense of self-efficacy in the use of basic ICT has been reported as 
linked to greater use of ICT in the classroom (Hatlevik 2016; Law et al. 2008). In ICILS 2013, 
teacher ICT self-efficacy was the teacher-level variable that showed the strongest association 
with teachers’ reported emphasis on developing students CIL, and “teachers who were confident 
about their own ICT capability were more likely than their less-confident colleagues to place a 
greater degree of emphasis on developing their students’ |CT-related skills” (Fraillon et al. 2014, 
p. 217). Furthermore ICILS 2013 reported that “older teachers typically held less positive views 
than younger teachers about using ICT and expressed lower confidence in their ability to use ICT 
in their teaching practice” (Fraillon et al. 2014, p. 257).The ICILS 2018 teacher questionnaire will 
therefore collect information on the background of teaching staff (such as age, gender, subject 
taught at school) and on their ICT experience (number of years using ICT for teaching purposes, 
general use of computers at different locations, participation in |CT-related professional 
development activities, and perceived self-confidence in using ICT for different tasks). 
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Teachers will also be asked to give theirviews on the positive and negative conseque 


ICT for teaching and learning, and to identify any factors that they think impede the use of ICT 


WORK 


nces of using 


for teaching and learning at their school. Results from ICILS 2013 indicated that teachers across 
participating countries tended to recognize positive benefits from using ICT in teaching (Fraillon 
et al. 2014). 

SITES 2006 findings suggested that ICT use by science and mathematics teachers is influenced 
by the school principal’s views about its value, as well as the ICT-related support teachers have 
at hand (Law et al. 2008). Findings also indicated that |CT-related teaching and learning can be 
constrained or facilitated by the school’s stated curriculum and its policies with regard to ICT. 
The ICILS school principal questionnaire will therefore collect data on the following factors: 

e The extent to which the school has policies and procedures relating to ICT use 

e The extent to which the school prioritizes ICT acquisition and resourcing 

e Perception of the importance ascribed to ICT use in teaching at the school 

e School-level expectations for teachers’ knowledge of and skills in using ICT 

e The extent to which teachers participate in ICT-related professional development. 
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have learned at school about different ICT-re 
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Process-related variables related to the home environment 


Home environment factors that potentially influence the learning process include the use of 
ICT in the home context and learning through interaction with family members. The student 
questionnaire therefore includes questions about the extent to which students have learned 
about different aspects of ICT use from family and/or friends, and how often they use ICT at 
outside of school (including at home). 


4.3.4 Individual context 


Antecedent variables at the individual level 


Antecedent variables at the level of the individual student consist of basic background 
characteristics that may influence students’ ClL-related knowledge and skills. Relevant factors in 
this category are age, gender, and educational aspirations. 


Students’ knowledge and skills in different learning areas tend to increase with age. However, 
cross-national data from grade-based surveys tend to find negative associations between age 
and achievement within a given grade-level within some countries (see, for example, Schulz et al. 
2017, p. 63). Findings from ICILS 2013 (Fraillon et al. 2014) showed a similar negative association 
which could be due to retention and progression policies where older students in the same grade 
(grade 8 for ICILS) are also those with lower achievement. 


Studies on educational achievement in numerous learning areas have found considerable 
differences between gender groups. In particular, cross-national research on reading literacy 
has shown larger gender differences in favor of females (Mullis et al. 2017; OECD 2016c). Males 
have traditionally tended to be somewhat more proficient in mathematics and science, but there 
is some evidence of a declining or non-existent gap at present (Martin et al. 2016; Mullis et al. 
2016; OECD 2016c). Data reported from Australian and US national assessments of |CT-related 
skills show significantly higher levels of achievement for female students when compared to male 
students (ACARA 2015; US Department of Education, National Center for Education Statistics 
2016). Cross-national results from ICILS 2013 also indicated that female students tended to 
have higher levels of CIL than their male counterparts (Fraillon et al. 2014). With regard to 
CT skills, however, some research suggests that the opposite relationship may be expected 
Atmatzidou and Demetriadis 2016). 


ndividual aspirations with regard to education provide an indication of students’ belief in their 
capacity to succeed in education and should be taken into account during any analysis of variation 
in students’ CIL/CT. ICILS 2013 results showed that students who expected to complete a 
university degree also had higher levels of CIL (Fraillon et al. 2014). The ICILS 2018 student 
questionnaire includes the same question as in the previous cycle to gauge students’ expected 
highest level of educational qualification. 


Process-related variables at the individual level 


Process-related variables at the individual level in this context include attitudinal, as well as 
behavioral factors. An individual’s self-beliefs regarding their own ability with respect to a certain 
learning area are often viewed as central to the process of learning, and are likely to have a 
reciprocal association with knowledge and skills (see for example, Schdber et al. 2018; Talsma et 
al. 2018). Furthermore, it is also important to include student perceptions about responsible and 
appropriate use of ICT, which can also be seen as intended learning outcomes from teaching CIL 
and CT. Behavioral variables also relate to using ICT for different purposes and needs, especially 
in terms of the potential that frequent and varied use of these tools has for facilitating student 
learning. 


The student questionnaire includes items designed to measure the extent to which students 
express confidence in doing a range of |CT-related tasks. According to Bandura (1993), students’ 
confidence in their ability to carry out specific tasks in an area (self-efficacy) is strongly associated 
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with their performance, as well as perseverance, emotions, and later study or career choices. 
oos and Azevedo (2009) concluded from their review of research on computer self-efficacy 
that this variable plays an integral role in learning in computer-based learning environments. The 
two authors examined factors related to computer self-efficacy and the relationships between 
computer self-efficacy, learning outcomes, and learning processes. They found a number of 
positive associations between behavioral and psychological factors, and computer self-efficacy.A 
particular finding was that students who experience behavioral modeling also report significantly 
higher computer self-efficacy than do students who experience more traditional instruction 
methods. 


nICILS 2013, two dimensions of self-efficacy were identified, one related to student confidence 
in undertaking basic |CT tasks (such as searching and finding a fileona computer) and another one 
reflecting confidence in more advanced tasks (such as creating a database, computer program 
or macro) (Schulz and Friedman 2015). While self-efficacy related to basic tasks tended to be 
positively correlated with CIL, confidence in undertaking advanced tasks was not consistently 
associated with students’ CIL (Fraillon et al. 2014). The ICILS 2018 includes a modified set of 
items measuring both student confidence in basic and more advanced ICT tasks that will be 
analyzed with regard to CIL/CT achievement. 


Applying ICT for different purposes on a regular basis has considerable potential to increase 
knowledge and skills in this area (see, for example, ACARA 2015; Fletcher et al. 2012: US 
Department of Education, National Center for Education Statistics 2016), and ICILS 2013 
showed frequent use of ICT for a wide range of activities (Fraillon et al. 2014). The ICILS 2018 
student questionnaire consequently includes questions (modified from the previous cycle) about 
the frequency of using different ICT applications, using the internet for social communication, 
and using ICT for recreational (leisure) activities. 


Data from other studies suggest a positive association between attitudes towards using ICT and 
academic achievement (Petko et al. 2016). In ICILS 2018, the student questionnaire includes a 
series of questions on students’ perceptions of the impact on ICT on society and whether they 
intend to use ICT in the future for work and study purposes. 


To gauge the educational context for the acquisition of CT skills, the ICILS 2018 student 
questionnaire asks students whether they study a CT-related subject (e.g., computing, computer 
science, information technology, informatics, or similar) in their current school year. This question 
is part of the international CT option. 
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CHAPTER 5 


ICILS instruments 


5.1. Test instrument overview 


The ICILS test is designed to provide students with an authentic computer-based assessment 
experience, balanced with the necessary contextual and functional restrictions to ensure that the 
tests are delivered in a uniform and fair way. ICILS uses a customized assessment platform that 
delivers the assessment content to students off-line (in the majority of schools the assessment is 
delivered from a USB drive). In order to maximize the authenticity of the assessment experience, 
the instrument uses purpose-built applications that use standard interface conventions. Students 
complete a variety of tasks, including multiple-choice and short text response questions, skills- 
based tasks, and information literacy and communication tasks, using a range of productivity 
software tools (such as text editors or presentation applications) and web-content. The web- 
content was developed for exclusive use in ICILS and was the only web-content available to 
students while they were completing the test. The purpose-built applications are designed to 
be consistent with the applications that can reasonably be expected to be within the realm of 
students’ typical experience of computer use. Students need to be able to both navigate the 
mechanics of the test and complete tasks presented to them. In order to support these two 
purposes, the test environment comprises two functional spaces: the test interface and the 
stimulus area (Figure 5.1). 


Figure 5.1: Test environment comprised of two functional spaces 
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5.1.1 Test interface 


The test interface serves a number of purposes. Firstly, it provides students with information 
about their progress through the test (such as the number of tasks completed and remaining, 
and the time remaining). The text for each task is provided at the bottom of the interface. This 
text may take the form of a question to be answered (in which case the answer space is also 
included in the section) or an instruction relating to the execution of one or more skills. The 
test interface includes navigation controls that allow students to move between tasks, and an 
information button that allows students to access general test-taking information and task- 
specific information, such as scoring criteria or detailed task instructions. The test interface also 
houses the stimulus area (see Figure 5.1). The stimulus area is a space that contains either non- 
interactive content, such as an image of a login screen for a website, or interactive content, such 
as electronic documents or live software applications. The test interface and stimulus area were 
similar to that used in ICILS 2013, but their appearance was modernized for ICILS 2018. The 
position and functionality of the elements in the test interface (such as the navigation button and 
task progress indicator) were not changed, but the appearance of the elements was modernized 
to be consistent with 2018 interface design conventions. 


5.2 The ICILS test instrument design 


[he ICILS test instrument consists of two tasks that are each delivered as 30-minute modules. 
n total, there are five CIL test modules in ICILS 2018. Three modules were developed and 
used in ICILS 2013, and these were kept secure to enable the establishment of trends in future 
cycles of ICILS. Two new modules were developed for the ICILS 2018 CIL test instrument. 
These new modules were designed to address contemporary relevant thematic content and 
software environments. Data collected from all five modules in ICILS 2018 are used as the basis 
for reporting ICILS 2018 CIL test results on the ICILS CIL achievement scale established for 
ICILS 2013. All students complete two of the five available CIL modules in a balanced rotation. 
The rotated module design for the modules enables the instrument to contain and consequently 
report on achievement against a larger amount of content (covering the breadth of the CIL 
framework and a range of difficulties) than any single student could reasonably complete in 60 
minutes. 


Two 25-minute test modules were developed forthe ICILS 2018 CT test. In countries participating 
in the CT international option, students complete the two CT modules (in randomized order) 
after completing the international CIL test and ICILS student questionnaire. 


5.2.1 ClL test modules 


A CIL test module is a sequence of tasks contextualized by a real-world theme and driven by 
a plausible narrative. Each module has a series of five to eight smaller tasks, each of which 
typically takes students less than one minute to complete, and each of which contributes to the 
development of contextual knowledge that underpins work ona single large task. The large tasks 
typically take 15-20 minutes to complete and involve the development of an information product 
(such as a presentation, poster, website or social media post) that makes use of information and 
resources managed by students in the lead-up tasks. The large tasks are specified for students in 
terms of the software tool and format to be used (and consequently the format of the product), 
the communicative purpose, and the target audience of the information product. Students are 
also provided with information about the criteria that will be used to assess each large task. 


The module themes are selected to be engaging and relevant to students, and the tasks are 
developed with a view to preventing prior content knowledge relating to a module theme from 
advantaging subgroups of students. This is achieved in three main ways: (1) by ensuring that all 
contextual information students need to manage the tasks is provided to students within the 
tasks; (2) by confirming that any technical (such as scientific) information used in modules is no 
more complex than the level of understanding typically expected of students in upper-primary/ 
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elementary school; and (3) by preventing students from returning to earlier tasks in a module, as 
information in a subsequent task could be used to answer a previous task (see Fraillon 2018 for 
a detailed explanation of these design features). 


The CIL module themes are all contextualized within a school environment, but they do not 
necessarily relate to conventional academic school work. For example, while modules may 
relate to communicating information about an aspect of science, social or environmental issues, 
modules can also relate to planning a class excursion or an online interest club with a community 
and social focus rather than academic focus. 


5.2.2 CT test modules 


The CT construct comprises two strands: conceptualizing problems and operationalizing solutions. 
Each of the two CT test modules focuses on one of these strands. Each module has a unifying 
theme and comprises a sequence of tasks that relate to the theme but, unlike the ClL modules, the 
tasks within the CT modules do not directly relate to the development of a large task. The tasks in 
the CT module focusing on conceptualizing problems related to planning aspects of a program to 
operate a driverless bus. This includes visual representation of real-world situations in ways that 
an support the development of computer programs to execute automated solutions. Examples 
of these are path diagrams, flow charts, and decision trees. Further tasks relate to the use of 
simulations to collect data and draw conclusions about real-world situations that can inform 
planning the development of a computer program. 


QO 


In the module focusing on operationalizing solutions, students work within a simple visual coding 
environment to create, test and debug code (blocks of code that have some specified and some 
configurable functions) that controls the actions of a drone used in a farming context. In this 
module, the tasks are incrementally more difficult as the students advance through the module. 
The difficulties of the tasks relate to the variety of code functions that are available and the 
complexity of the sequence of actions required by the drone for completion of the task. 


5.2.3. Test module rotation 


Each student completes two of the five available CIL test modules. These modules are allocated 
to students in a balanced randomized design. There are 20 possible permutations of the two 
CIL modules selected from the five available modules. Each student is randomly allocated one 
module permutation. 


In countries participating in the ICILS 2018 CT option, students complete the two CT test modules 
after having finished both the CIL test and student questionnaire. There are two permutations of 
the two CT modules and each student is randomly allocated one module permutation. 


5.3 Types of assessment task: CIL 


The computer-based assessment of CIL contains three types of task that are integrated into a 
single testing environment. This section contains details of each of these tasks with illustrative 
example®. Some of the example tasks are from the ICILS 2013 module “After-school exercise,” 
where the student's central task was to design a poster to promote an after-school exercise 
program. Other example tasks are taken from a demonstration module created to illustrate 
some task formats that are otherwise only part of the secure ICILS assessment materials. This 
demonstration module is based on the idea of students working with a group of collaborators to 
plan the design of a new garden area in their school. In this module, students had to prepare an 
information sheet that explained and engendered support for their garden design, intended to 
encourage their classmates to vote for their design. 


5 At the time of publication of this framework, most ICILS test tasks are secure. Where released modules do not include a 
particular task type, illustrative examples have been created for use in this framework to accurately represent the types 
of task formats and content materials used in ICILS. 
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5.3.1 Task type 1: Information-based response tasks 


Information-based response tasks use a digital interface to deliver pencil-and-paper style 
questions ina slightly richer format than traditional paper-based methods. The stimulus material 
is typically anon-interactive representation of a computer-based problem or information source. 
The response formats for these tasks may be multiple choice, constructed-response, or drag- 
and-drop ones that use the technology only to display the stimulus material and record student 
responses. In these tasks, the computer-based environment is used to capture evidence of 
students’ knowledge and understanding of ClLindependently of students using anything beyond 
the most basic skills required to record a response. 


As illustrations of an information-based response task format, example task 1 (Figure 5.2) 
requires students to examine four organizational-structure diagrams for a website and to 
select the structure that best suits a given set of six pages of content. This relates to Aspect 
2.2 (managing information) of the CIL construct. Similarly, example task 2 (Figure 5.3) requires 
students to examine a non-interactive email (in this case a suspicious phishing email) and to 
respond using free text in a text entry box in the lower section of the test interface. 


Figure 5.2: Example task 1 (a typical multiple-choice task) 
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The dynamic computer-based environment in example task 1 (Figure 5.2) enables students to 
view each of the four website structures in turn. The stimulus could also be presented in a static 
form (i.e., showing all four diagrams together) in a pencil-and-paper test. The simplest multiple- 
choice tasks in ICILS could also be presented in an equivalent form on paper. 


However, because example task 1 allows students to drag and drop the web page contents 
into each organizational-structure template and thereby “try out” the different information 
structures in order to support their choice of the best structure; the computer-based stimulus 
facet of this task extends beyond what could be made easily available in a pencil-and-paper 
format. The task then enables students to provide their answer through a conventional multiple- 
choice format (shown in the lower area of the test interface), with one correct response that 
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can be automatically scored. While the drag-and-drop functionality in example task 1 serves 
as an aid to determine the correct response, in other ICILS tasks this functionality serves as a 
method for recording student responses. The ICILS assessment uses the drag-and-drop task 
format whenever students are required to classify information into groups or to match objects 
or concepts according to their characteristics. 


Figure 5.3: Example task 2 (open-response task from After-school exercise: ICILS 2013 test 
interface) 
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The stimulus material in example task 2 (Figure 5.3) from “After-school exercise” contains 
a phishing email with some metadata, such as the email address of the sender and the URL 
associated with the anchor text link. The task is presented to students as an example of an 
email that is trying to trick the user into clicking the link. Students must identify the discrepancy 
between the fictionally branded domain name “WebDocs’” in the URL at the bottom of the email 
and the domain name associated with the sender’s email address (freemail). Example task 2 
relates to Aspect 4.2 (using information responsibly and safely) of the CIL construct. Responses 
to this task are recorded as text and scored by scorers according to a pre-defined scoring guide. 


5.3.2 Task type 2: Skills tasks 


Skills tasks require students to use interactive simulations of generic software or universal 
applications to complete an action. These may be single-action tasks (such as copying, pasting, or 
selecting a browser tab) or may contain a sequence of steps (such as “Save As” with a specific file 
name, or navigation through a menu structure). The tasks are designed to allow for all possible 
“correct” pathways for completing a task (such as using keyboard shortcuts or menu items) and 
the response data are recorded by the testing software. Some skills tasks only require students 
to execute given software commands, while others require students to execute commands along 
with some information processing. Skills tasks are scored automatically. 
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The ICILS student test contains linear and nonlinear skills tasks. A linear skills task may be as 
simple as executing a single command (such as opening a file from the desktop), or may require 
more than one step to complete the task. All appropriate methods of executing a command (e.g., 
using the mouse, pull-down menus, or keyboard shortcuts) are scored as equivalent and correct. 
Linear skills tasks that require the execution of more than one command can only be completed 
correctly if the commands are executed in a necessary prescribed sequence. For example, if 
students are instructed to copy and paste an image, they would first need to select the image and 
then execute the copy and paste commands in that order. Responses are automatically detected 
and scored once participants have reached an “endpoint” to a task. 


Figure 5.4: Example task 3 (linear skills task from After-school exercise: ICILS 2013 test interface) 
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Example task 3 (Figure 5.4) provides an example of a linear skills task that requires students to 
change the settings for a document in a collaborative workspace in order to permit edit access to 
specified people. Students must first click on the settings/sharing menu link and then enter the 
specified username into a field. Example task 3 relates to aspect 1.2 (computer use conventions) 
of the CIL construct. 


Nonlinear skills tasks require students to execute a software command (or reach a desired 
outcome) by executing subcommands in a number of different sequences. Example task 4 is one 
such nonlinear skills task (Figure 5.5). This task requires students to use the filtering functions 
of a web-based database and to interpret some simple text in order to locate an object (a plant) 
that matches a given set of characteristics. The task is thus an example of a nonlinear skills task 
that requires information-processing skills and relates to Aspect 2.2 (managing information) of 
the CIL construct. The web-based database contains too many objects for a student to search 
manually with ease. As such, the automatic scoring gives the highest level of credit to students 
who make use of the filtering functions (in any order) to support their search. Students who 
identify the correct task without using the filters receive less credit. 


ICILS INSTRUMENTS 


Figure 5.5: Example task 4 (nonlinear skills task) 
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5.3.3 Task 
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ven commands. This approach may require students to use multiple applications 


such as email applications, web pages, spreadsheets, and word processing 
software) as is typically required when using computer software to perform 
plex tasks. Each student’s work is automatically saved as an information product 
t by scorers according to a prescribed set of criteria. 


Example task 5 (Figure 5.6) illustrates a simple authoring task, which requires students to use a 


basic map-drawing application 


the plan. This 


is a simple authoring task bec 


to create a garden design plan that represents the text describing 
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of software (the mapping software) to complete the task. It is also simple because there is a 
relatively narrow range of “correct” ways in which the student can draw the garden design to 
match the specifications. The task is manually scored according to the accuracy with which the 
different specified elements of the garden design are shown in the diagram. 
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Figure 5.6: Example task 5 (simple authoring task) 
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Example task 6 is a more complex authoring task (Figure 5.7), requiring students to use 
information from a website across three sub-pages to create a poster that promotes an after- 
school exercise program. Students must use information from a range of electronic sources to 
create an information sheet that explains and promotes their garden design. The stimulus is 
nonlinear, fully interactive, and behaves intuitively. Students can navigate between browser tabs 
to switch between the website and the poster design web application. They can copy and paste 
text from the website to the poster software, and insert elements, such as images and text, on 
to the canvas of the poster design application. The final information product is saved, stored, 
and then scored against a set of criteria. The scoring criteria can be categorized as relating to 
students’ use of (1) the software features and (2) the available information. 


Criteria relating to students’ use of software features can include their ability to use color, text 
formatting, and general page layout. These criteria typically have an internal hierarchy based on 
the degree to which the software features are used to support or even enhance the communicative 
effect of the information product. Criteria relating to students’ use of information can also include 
students’ adaptation of information, the relevance (and accuracy) of information selected for and 
used in the information product, and the appropriateness of selected information for the target 
audience. Note that the use of information is only assessed with respect to students’ use of the 
information provided to them for use in the module. The highest level of credit is given to student 
work that demonstrates ability to use the software features to enhance the communicative effect 
of the information product. The lowest level of credit is given to pieces of work that show no 
application of the relevant software feature, or uncontrolled use (such as extremely poor color 
contrast or overlapping text) that inhibits comprehension of the product. The range of criteria 
available to evaluate example task 6 means that the single task collects evidence of student 
achievement relating to aspects 3.1 (transforming information) and 3.2 (creating information) 
of the CIL construct. 
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Figure 5.7: Example task 6 (complex authoring task from After-school exercise: ICILS 2013 test 
interface) 
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5.4 Types of assessment task: CT 


Like the CIL test, the CT assessment also contains information-based response tasks and 
nonlinear skills tasks (as described in section 5.3). However, in addition to these, the CT 
assessment instrument includes task types that are unique to the CT assessment. We created 
illustrative examples of these tasks specifically for inclusion in this framework®. These examples 
are similar to the tasks developed for use in each of the two CT test modules. 


5.4.1 Task type 4: Nonlinear systems transfer tasks 


Nonlinear systems transfer tasks require students to interpret, transfer and adapt algorithmic 
information so that the outcomes of the application of algorithmic instructions can be displayed 
visually. Example task 7 (Figure 5.8) requires students to follow the steps of a simple algorithm 
(left panel) and to transfer and adapt the steps of the algorithm to a visual display of their 
application (right panel). In successfully completing this process, students demonstrate both 
understanding of the visual system (including visual feedback) and the steps of the algorithm. 
This task relates to aspect 1.3 (collecting and representing relevant data) of the CT construct. 
These types of tasks are nonlinear because the student can select any node connected by the 
gray line in the right panel in any order, irrespective of what is described in the left panel. 


6 At the time of publication of this framework, all ICILS 2018 CT items are secure. Illustrative examples have been created 
for use in this framework to accurately represent the types of task formats and content materials used in the ICILS 
computational thinking modules. 
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Figure 5.8: Example task 7 (nonlinear systems transfer task) 
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The student does not receive feedback about the correctness of their response but does receive 
feedback that their choice was registered. When the node selection is made, the color of the 
node changes from blue to red, and the line between the nodes changes from gray to red with 
an arrow indicating the direction of the movement between the nodes. These types of tasks are 
systems transfer tasks because they require the student to decode information presented in 
one system, deconstruct the rules of a second system, and adapt the information for transfer 
between the two systems. 


5.4.2 Task type 5: Simulation tasks 


Simulation tasks require students to set parameters, run a simulation to collect data, and 
interpret the data to answer a research question. Example task 8 (Figure 5.9) requires students 
to configure the simulation tool and run simulations to identify a drone’s optimal flight path over 
a set of pumpkins. 


The decision tree (see left panel of Figure 5.9) is used to configure the flight path of the drone 
and the starting positon of the drone is also configured by the student. The student can then 
vary the configuration and run the simulation to identify the optimal parameters for the specified 
purpose. Simulation tasks such as example task 8 typically relate to aspect 1.3 (collecting and 
representing relevant data) of the CT construct. 


5.4.3 Task type 6: Visual coding tasks 


The ICILS 2018 CT test included a set of tasks that made use of a visual coding environment. 
The task interface and tasks are secure and examples of the interface and tasks cannot be shown 
without compromising the security of the test. Consequently, in this section, we provide only a 
description of the properties of the visual coding task interface and the types of visual coding 
tasks students may complete in the ICILS CT test. 
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Figure 5.9: Example task 8 (Simulation task) 
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Visual coding test environment 


The visual coding test environment uses the ICILS test interface. However, in the ICILS 2018 
visual coding test module, students were permitted to return to previous tasks within the module. 
This decision was made because, unlike within other ICILS test modules, the visual coding tasks 
did not follow a sequence where information provided in later tasks could potentially reveal the 
answer to earlier tasks. Consequently the visual coding test interface included the facility for 
students to “flag” tasks that they might wish to return to, and a navigation feature that allowed 
them to navigate freely between tasks they had already viewed. 


The overarching objective for the visual coding environment was for students to complete coding 
tasks relating to the function of a drone used in farming. The visual coding test environment 
included the following key elements: 


e Awork space in which code blocks could be placed, ordered and re-ordered, and removed 
from the work space 


e Aspace containing the code blocks that could be selected and used in the work space. These 
included code blocks controlling movement of the drone, some simple configurable commands 
for the drone to execute, simple loops, and conditional statements 

e The facility for students to execute the code at any time and to see the consequent behavior of 
the drone as the code was being executed 

e The facility to reset the code in the work space (to the default state of each task) and to reset 
the starting position of the drone before executing code. 
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Algorithm construction tasks 


Algorithm construction tasks require students to develop their own solution to a problem by 
iteratively adding code blocks to the work space and executing the algorithm to see the results. 
These tasks typically allow for a variety of solutions with differing complexity (variety of code 
blocks) and depth (the number of levels deep nested codes are executed). Student responses are 
scored with respect to the accuracy with which the code achieves the specified goal, as well as 
the efficiency of the code, taking into account the number of code blocks used and the students’ 
use of looping and conditional logic the algorithm. These tasks relate to aspect 2.2 (developing 
algorithms, programs and interfaces) of the ICILS CT framework. 


Algorithm debugging tasks 


Algorithm debugging tasks require students to modify an existing algorithm (configuration 
of code blocks in the work space) to solve the problem presented by the task instructions. In 
these tasks the students are presented with an existing set of code blocks in the work space, a 
description of the intended outcome of executing the code and an indication that the code is not 
working and needs to be corrected. Students can freely modify the code and also reset the code 
blocks in the workspace to the default state of the task (i.e., redisplaying the original incorrect 
code requiring debugging). Students are assessed on how closely their specified solution matches 
the ideal solution, including how accurately the code meets the specified goal. These tasks relate 
to aspect 2.1 (planning and evaluating solutions) of the ICILS CT framework. 


5.5 Mapping test items to the CIL and CT frameworks 


The test items that comprise the assessment modules are based on the strands and constituent 
aspects in the assessment framework (see section 2.5). The ClL and CT frameworks are central 
to the process of instrument development because they provide a theoretical underpinning 
for the assessment and a way of describing its content. The CIL test items were mapped to the 
assessment strands and the constituent strands in the CIL framework (Table 5.1) and CT test 
items were mapped to aspects and levels in CT test (Table 5.2). 


In the CIL test, more items and score points per item relate to Strand 2 and Strand 3 than the 
other strands of the CIL construct (see Table 5.1). The main reason for this is that the large tasks 
at the end of each module focus on students’ creation of an information product and therefore 
require each of these tasks to be assessed via multiple criteria with multiple score categories. 
Assessment of the large tasks focuses on Aspects 3.1 and 3.2, and together these contribute the 
largest number of associated score points across the test modules. The test design of ICILS was 
not planned to assess equal proportions of all aspects of the ClL construct, but rather to ensure 
some coverage of all aspects as part of an authentic set of assessment activities in context. The 
balance of items and score points relating to the different aspects of the CIL construct reflects 
the balance of time that students are expected to spend completing the different tasks. 


While there is a similar number of items assessing each of the two CT strands, the number of 
score points available for strand 2 (operationalizing solutions) is approximately double those 
for strand 1 (see Table 5.2). Assessment of the quality of students’ operationalized solutions 
(typically their visual coding solutions to specified problems) are assessed using multiple criteria. 
The test design of ICILS was not planned to assess equal proportions of all aspects of the CT 
construct, but rather to ensure some coverage of all aspects as part of an authentic set of 
assessment activities in context. 
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Table 5.1: Mapping the CIL test items to the CIL framework 


CIL strand/aspect Total Maximum total 
(Items) (score points)* 

Strand 1: Understanding computer use 

Aspect 1.1: Foundations of computer use 2 WA 

Aspect 1.2: Computer use conventions 11 13 

Total (strand 1) 13 15 

Strand 2: Gathering information 

Aspect 2.1: Accessing and evaluating information 14 16 

Aspect 2.2: Managing information 8 8 

Total (strand 2) 22 24 

Strand 3: Producing information 

Aspect 3.1: Transforming information 15 20 

Aspect 3.2: Creating information 21 SL 

Total (strand 3) 36 51 

Strand 4: Digital communication 

Aspect 4.1: Sharing information d 8 

Aspect 4.2: Using information responsibly and safely 

Total (strand 4) 10 12 

Table 5.2: Mapping the CT test items to the CT framework 

CT strand/aspect Total Maximum total 
(Items) (score points)* 

Strand 1: Conceptualizing problems 

Aspect 1.1: Knowing about and understanding digital systems 3 4 

Aspect 1.2: Formulating and analyzing problems 2 

Aspect 1.3: Collecting and representing relevant data 3 5 

Total (strand 1) 8 16 

Strand 2: Operationalizing solutions 

Aspect 2.1: Planning and evaluating solutions 4 12 

Aspect 2.2: Developing algorithms, programs and interfaces 11 

Total (strand 2) 8 23 
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5.6 The ICILS student questionnaire and context instruments 


5.6.1 Student questionnaire 


The student questionnaire is based on the review of previous research discussed as part of the 
contextual framework (see Chapter 4), and was designed primarily to collect data that address 
Research Questions 3 and 4 for both CIL and CT: 


RQ3 How do students’ levels of access to, familiarity with, and self-reported proficiency in using 
computers relate to students’ CIL and CT? 


RQ4 What aspects of students’ personal and social backgrounds (such as gender, and 
socioeconomic background) are related to students’ CIL and CT? 


Data gathered from the student questionnaire are used for two purposes. Firstly, these data are 
used in analyses that examine the relationships between student-level factors and measured 
CIL and CT. Secondly, these data are used to provide descriptive information about patterns of 
computer access and use across and within countries. 


The student questionnaire is designed to provide the following indices regarding student and 
home background: 


e Students’ age (in years) 


e Students’ gender 


e Students’ expected highest level of educational qualifications 
e Students’ immigrant background 

e Students’ language use at home (test language or others) 

e Students’ parents’ highest occupational status 

e Students’ parents’ highest level of education 

e Student reports on home literacy (number of books at home) 


e Student reports on ICT resources at home 


e Students’ experience with ICT. 


The student questionnaire contains questions to generate data reflecting the following aspects 
of ICT use and attitudes related to ICT: 


e Students’ experience in using ICT (frequency) 
e Students’ use of ICT applications (frequency) 
e Students’ use of ICT for social communication (frequency) 


i 


e Students’ use of ICT for exchanging information (frequency) 


e Students’ use of ICT for accessing online content (frequency) 


e Students’ use of ICT for recreational purposes (frequency) 


i£ 


e Students’ use of ICT for school-related purposes (frequency) 


e Students’ use of ICT in school subject lessons (frequency) 


e Students’ use of ICT tools in class (frequency) 


e Student reports on learning of ICT tasks at school 


e Students’ ICT self-efficacy 


e Student reports on learning about responsible ICT use at school 


e Students’ perceptions about the impact of ICT for society 


e Students’ expectations of future ICT use for work and study 


e Student reports on the extent of learning about approaches to computational thinking at 
school. 
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5.6.2 Teacher questionnaire 


The teacher questionnaire is largely concerned with information about teachers’ perceptions 
of ICT in schools and their use of ICT in educational activities in their teaching. Together with 
questionnaires completed by the school principal and ICT coordinator, the teacher questionnaire 
is based on the contextual framework (Chapter 4) and designed to collect data that address 
Research Question 2 for both CIL and CT: 


RQ2 What aspects of schools and countries are related to students’ CIL and CT? 


The assumption is that the extent to which ICT is used in schools, and the ways in which ICT is 
used in schools to teach information literacy, will impact on the development of students’ CIL 
and CT. Information from the teacher questionnaire will also be used to describe the use of ICT 
in pedagogy among countries and major teaching areas. 


It will not be possible to link teacher-based information to individual students. Rather, that 
information can be used to generate school-level indicators for potential two-level regression 
analyses in conjunction with student-based data. 


The population for the ICILS teacher survey is defined as all teachers teaching regular school 
subjects to the students in the target grade (generally grade 8) at each sampled school. Fifteen 
teachers are selected at random from all teachers teaching the target grade at each sampled 
school to complete the teacher survey’. This cluster size is required to produce: 


e School-level estimates with sufficient precision to be used inanalyses that examine associations 
with student outcomes 


e Population estimates with precision similar to those generated from student data. 


The teacher questionnaire consists of questions about teachers’ backgrounds, their familiarity 
with ICT, their use of ICT in teaching a reference class, their perceptions of ICT at school, and 
their training to use ICT in teaching. 


The teacher questionnaire is designed to generate data reflecting the following aspects of 
teacher perceptions regarding ICT and its use in education: 


e Teachers’ CIL self-efficacy 


e Teachers’ use of ICT tools in class 


e Teacher reports on student engagement in learning activities with ICT 
e Teacher reports on use of ICT in teaching and learning practices 
e Teachers’ perceptions of the emphasis on ICT skills development in class 


e Teachers’ positive views regarding the use of ICT in teaching and learning 


e Teachers’ negative views regarding the use of ICT in teaching and learning 


e Teachers’ perceptions of adequacy of resources at their school 


e Teachers’ participation in ICT professional development 


e Teachers’ perceptions of collaboration for ICT use 


e Teachers’ emphases on teaching approaches to computational thinking in class. 


7 \In small schools this means all teachers of grade 8 students. 
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5.6.3 School questionnaires 


There are two complementary school questionnaires: a principal questionnaire and an ICT 
coordinator questionnaire. The principal questionnaire focuses on characteristics of the school 
and broad policies, procedures and priorities for ICT in the school. The coordinator questionnaire 
focuses on ICT resources in the school and the policies and practices that make use of those 
resources to support learning. While these two questionnaires should ideally be completed by 
different people, ICILS 2018 makes provision for the possibility that both may be completed by 
the same person in a small school where there is no identifiable ICT coordinator. 


Principal questionnaire 


The principal questionnaire is designed primarily to collect data that address Research Question 
2 for both CIL and CT: 


RQ2 What aspects of schools and countries are related to students’ CIL and CT? 


The assumption underlying Research Question 2 is that the extent to which, and manner in 
which ICT is used in schools will impact on the development of students’ CIL and CT. The ICILS 
framework assumes that the school principal can provide important perspectives on school 
practices and policies regarding the pedagogical use of ICT. 


The principal questionnaire also collects data that will provide further context on variation in the 
use of ICT in pedagogy across education systems. 


The ICILS principal questionnaire covers the following areas: characteristics of the principal 
including their use of ICT), school characteristics (number of enrollments, range of grades 
taught, characteristics of the school location, ratio of female to male enrollments), management 
of ICT in the school, encouragement to use ICT in teaching and learning, schools’ pedagogical 
orientations, and provision for professional development in the use of ICT. 


The principal questionnaire includes questions designed to collect data about the following 
contextual aspects at the school level: 


e School principals’ use of computers for school-related purposes (frequency) 


e School size (student enrollment) 
e Student-teacher ratio 


e School structure and management 


e Economic background of students 


e School principals’ perceptions of the importance of ICT use at school 


e School principal reports on their expectations of teachers’ ICT skills 


e School principal reports on ICT policies and procedures 


— 


e School principal reports on teachers’ professional development for ICT use 


— 


e School principal reports on school priorities for ICT use in teaching and learning. 


ICT coordinator questionnaire 


The ICT coordinator questionnaire will be designed primarily to collect data that address 
Research Question 2 for both CIL and CT: 


RQ2 What aspects of schools and countries are related to students’ CIL and CT? 


The assumption underlying Research Question 2 is that the extent to which, and manner in 
which ICT is used in schools will impact on the development of students’ CIL and CT. The ICILS 
framework assumes that the ICT coordinator can provide important perspectives on school 
practices and policies regarding the pedagogical use of ICT. The ICT coordinator questionnaire 
will also collect data that provide further context on variation in the use of ICT in pedagogy 
across education systems. 
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The ICT coordinator questionnaire collects data on ICT resources (numbers of computers of 
different types, availability of computers for student use, availability of other ICT devices, 
availability of digital learning resources, networking and internet connectivity), ICT use in 
the school (provision for specialist teaching of ICT, emphases in curriculum areas, learning 
management systems, school administration), ICT technical support (maintenance provision, 
support for managing resources), and provision for professional development in ICT at school. 


The ICT coordinator questionnaire includes questions designed to generate data reflecting the 
following !CT-related aspects: 


e School experience in using ICT 


e School policies towards the use of ICT at school 


e The computer-student ratio at school 


e The quality of ICT resources at school 


e Perceptions of hindrances to the use of ICT in teaching and learning at school 


e Perceptions of school emphasis on teaching activities to develop students’ CT skills. 


5.6.4 National contexts survey 


The national contexts survey is intended to collect data that primarily addresses Research 
Question 2 for both CIL and CT: 


RQ2 What aspects of schools and countries are related to students’ CIL and CT? 


The assumption underlying Research Question 2 is that the opportunities to use ICT impact on 
opportunity to learn about CIL and CT and therefore on the development of student outcomes 
in these domains. 


Data from the national contexts survey will be used to compare profiles of ClL and CT education 
in participating education systems. In addition, the data will provide information about contextual 
factors concerned with structure of the education system and other aspects of education policy 
or the analysis of differences in approaches to ICT-related learning across education systems. 
Data from the national context questionnaire will be used for two broad purposes: 


if 


e To provide systematic descriptions of policy and practice in the use of information and 
communication technologies in school education across participating ICILS countries 


e To provide systematic data that can be used as a basis for interpreting differences among 
education systems in |CT-related learning outcomes as well as patterns of relationships 
among factors that are related to ICT-related learning outcomes. 
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